ConnectME: connecting content for future TV & video


Published on

Today, media material is increasingly digital and shifting to delivery via IP and the Web,
including cultural artifacts or broadcast television. This opens up the possibility of new
services deriving added value from such material by combining it with other material
elsewhere on the Web which is related to it or enhances it in a meaningful way, to the
benefit of the owner of the original content, the providers of the content enhancing it and
the end consumer who can access and interact with these new services. Since the
services are built around providing new experiences through connecting different related
media together, we consider such services to be Connected Media Experiences

In particular, much rich depth of information and service functionality associated to
content in video is not derived today due to a lack of suitably granular description of
video, including linking of video objects to the concepts they represent. For example,
news reports about local tourism and events are not linked to tourist information and
event–related services which a viewer may (spontaneously) wish to access and make
use of.

The technological result of the ConnectME project will be an end-to-end service platform
to host those added-value services over different networks, providing the common
required functionality of each service: multimedia annotation and subsequent enrichment
with related content from the Web, combined with the packaging and delivery of
synchronized multimedia presentations to the end device. At the device, intuitive user
interfaces must be developed so that the selection of on-screen objects and the
browsing of the associated content can be done in a non-disruptive and intuitive fashion.
As a result, ConnectME facilitates a new interactive media experience built on top of the
convergence of TV/video and the Web.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • WireWax on iPad video:
  • Dev CMF:
    Prod CMF:
  • ConnectME videos: Login to - password: connectme. Currently there are 18 videos.
  • A sample mediaRSS file:
    What can be created in CMF as video metadata:
    (NOTES: we should be able to include video creator, owner, license...)
  • A sample TV Anytime file:
    What can be created in CMF as video metadata:
  • API:
    Doc: see
    accepts video url, title, description, keywords
    HOW TO INCLUDE OTHER METADATA, e.g. owner, creator, license
  • Getting started:
    Open video → get a list of videos from the CMF the tool is configured to use
    „Enter a video URL“ option from „Open video“ also uses the Web service to add a new video instance to the CMF
    From Yoovis, the tool is opened directly using a special URL, e.g.
    From PS Media, TBD
    If video is pre-annotated, these are loaded into the add new annotation interface
    If video is annotated, these are loaded into the timeline interface
  • Annotation interface:
    - enter a concept or thing uses DBPedia
    - enter a place uses GeoNames
    - choose a suggestion uses pre-annotations (if any)
  • Published at
  • TO BE CORRECTED; USING mao:locator
  • Both tools are STABLE and production is FROZEN
  • CMF can be configured to use different Stanbol enrichment chains
    - for ingestion via Web service
    the default is DBPedia entity extraction using a local DBPedia dump (subset) from the title, descriptions and keywords
    Same for subtitle files, but generates annotations NOT pre-annotations
    - for ingestion from Yoovis CMS (mediaRSS)
    The default is keyword extraction using a local specific taxonomy of SKOS concepts
    (, open
    Yoovis taxonomy looks completely flat, intended? Where are the mappings to LOD?
    I still need to see evidence of use of Stanbol giving us the best (pre-)annotations
    Online NER services (like NERD) could be used, NERD also supports subtitle files (SRT)
  • AFAIK no annotation views are programmed in any CMF
    (widgets encapsulate this functionality if required by SPARQL querying cached LOD)
    LSI integration has been made in the DEVELOPMENT CMF?
  • 1.
  • Currently only enabled on the test player at
  • ConnectME: connecting content for future TV & video

    1. 1. ConnectME: connecting content for future online TV & video Lyndon Nixon Last update: September 25,2013 More at Contact :
    2. 2. Connected Media Experiences • New services deriving added value for video by combining it with complementary content from the Web • Linking of video objects to the concepts they represent, and hence related content about those concepts • Provisioned as an enhanced in-video experience • Delivered via an end-to-end service platform hosting the added value services over different networks
    3. 3. Growth of Internet on the TV Below: Boxee, ConnectedTV, Samsung Smart TV (
    4. 4. Growth of TV on the Internet
    5. 5. Growth of mobile video Video streaming accounts for 37% of all mobile traffic Of all video streaming traffic, YouTube accounts for 45% A Cisco study* on mobile traffic growth expects • 66% of all traffic by 2014 will be video • having increased 66-fold from 2009 to 2014 * Cisco‘s Visual Networking Index Global Mobile Data Forecast 2010-2015
    6. 6. Trending towards connected media
    7. 7. Associating content to video
    8. 8. Emerging hypermedia Asterpix‘s Asterbot discontinued! Cf. Videoclix, Overlay.TV, Klickable, ClickVideo Wirewax
    9. 9. What could be a connected media experience? Where is Obertauern? Which peak is that Can I ski there? Ho What is the buildin If it is a hotel, how the accommodat
    10. 10. How to experience connected media?
    11. 11. The business case for hypervideo • Offering an added value service – Solely to win more visitors & keep them using the service – Encourage upgrades to premium packages (by content provider, by end consumer) • Monetarization of the links – E-commerce (drive traffic to electronic purchases) – Content partnerships (drive traffic to others‘ content) – Ad inserts
    12. 12. The technical case for connected media • Channel annotation, enrichment and delivery are all services of the ConnectME framework
    13. 13. Content workflow • Upload video in partner CMS • Fill in basic information and „commit“ video to the ConnectME Framework (CMF) • CMF includes video instance and preannotates it from the basic information and video analysis components • Annotation tool provides preview of preannotation and means to finalise annotation • CMF handles links to content enrichments • Preview in ConnectME player with widgets rendering different content enrichments
    14. 14. Relationship Linked media framework (LMF) <-> Connected media framework (CMF) • CMF wraps the LMF reusing its inherent media & metadata handling functionality and realising the ConnectME workflow on top. Analysis, annotation Visual analysis LMF Speech to text Workflow Linked Media Store Concept extraction REGISTRATION Interlinking, enrichment Annotation views ConnectME ontology based Linked Media engine rul es LOD REST Annotation tool Interlinked media generation Presentation engine Enrichment playout Presentation, playout REST AUTENTICATION Hypervideo browser
    15. 15. Channel annotation • Grouping of videos into a „channel“ • Easy to use tools for content owners or hosts to annotate video segments with links to semantic concepts – Manual annotation tool – Supported by • automatic classifiers over visual features, audio, subtitling/transcripts, or other annotations incl. crowdsourced tagging • tool tips for choosing concepts from pre-selected sources
    16. 16. Yoovis use case: ConnectME channel Courtesy Yoovis GmbH,
    17. 17. Integration of annotation
    18. 18. Pre-annotation in Yoovis use case • Yoovis CMS calls a Web service on CMF which accepts the mediaRSS format – „registers“ a video instance in the CMF – Includes video metadata directly extractable from the mediaRSS, e.g. locator(s) – Generates video annotations from title, description and tags from the author • Attachs these „pre-annotations“ as 'dct:subjects' in the video metadata with local Yoovis terms, mapped where possible to LOD such as DBPedia instances
    19. 19. Pre-annotation in the PS Media use case • PS Media CMS pushes to CMF a RDF description generated in Drupal – RDF Module maps Drupal node data to RDF – LMF Module pushes this RDF into our server – Generates video annotations from title, description and subtitles from the EPG • Attachs the „pre-annotations“ as in Yoovis use case • Entity extraction from subtitles can associate entities already to a temporal segment (store as annotation)
    20. 20. Pre-annotation (generic) • Any client could directly call a Web service on CMF which takes the parameters title, description, URL & subtitles – „registers“ a video instance in the CMF – Includes locator(s) from URL parameter – Generates video annotations from title, description and subtitles • Attachs the „pre-annotations“ as in Yoovis use case • Entity extraction from subtitles can associate entities already to a temporal segment (store as annotation)
    21. 21. Annotation suite - HTML 5, CSS 3, responsive layout, based on Twitter Bootstrap - jQuery, JS based UI - Timeline based interface - Allows adding/editing/ deleting annotations in/to the CMF - Allows registering videos to and loading video annotations from the CMF - Multilanguage support (currently EN, DE) - Access via OpenID
    22. 22. Annotation suite - Annotations are RDF and concepts are using LOD identifiers - Annotation export in N3, RDF/XML and Turtle - Manual annotation with LOD concepts (URIs) and HTML content (URLs) - Currently Plugins for dbpedia and geonames search APIs - Plugin based architecture allows adding further LOD sources easily
    23. 23. Annotation model published at Re-uses a subset of W3C Media Ontology and Open Annotation Model Extends mao:hasKeyword to subproperties of explicitly/implicitly seeing/mentioning concepts. Pre-annotations attached to video using dct:subject and skos:Concepts cmo:preferredLabel overrides concept labels cmo:hasContent points to static Web content
    24. 24. ConnectME annotation instance < > a ma:MediaResource; ma:hasSubtitling < >; ma:hasLanguage [ rdfs:label "de" ] ; ma:hasContributor <>; ma:creator http:// ma:date "2011-09-13"^^xsd:date; ma:hasRelatedLocation "27.14, 16.83"^^geo:point; ma:isCopyrightedBy <> ; ma:hasPolicy < ...>; ma:hasFragment http:// a ma:MediaFragment; ma:locator < >; cma:explicitlyMentions < >. <> a oac:Annotation; oac:target <>; oac:body < >; dcterms:creator http://; dcterms:created "2013-0219T16:20:38+00:00"^^xsd:dateTime .
    25. 25. Annotation suite Production annotation suite
    26. 26. Annotation via the CMF • Video metadata is turned into ConnectME annotations on the CMF – Ingestion services generate top level metadata such as video title, description, author, license... • Analysis of textual metadata to extract keywords a task of the Apache Stanbol Enhancer – A generic enrichment chain for entities->DBPedia – Specific enrichment chains for Yoovis and PS Media (entities->keywords(->LOD))
    27. 27. Enrichment via the CMF • Each concept needs to be linked to related content items – The player widgets could query this directly but – The CMF can cache relevant links so that the player can make simple queries for content • Common data model for links regardless of source, type etc. – See the Media Ontology for Linked Media
    28. 28. Direct & indirect links Direct links in the concept metadata to related content can be transformed to the common model by extending CMF reasoning rules („Annotation views“) ($c $m) -> ($m $c) - Indirect links must be derived for types of concept using other Web based content sources (APIs)
    29. 29. Linked Services Infrastructure • – Web interface and REST API – Input: a LOD URI, output: RDF for matching media resources
    30. 30. Linked Services Infrastructure Middleware to different Web APIs (YouTube, Flickr, Ookaboo, Foursquare Venues, Instagram) » „lowers“ the input concept to query parameters such as label, long/lat ... » „lifts“ the API response to RDF links for the CMF, where a content item is described and connected to the original concept by ma:hasKeyword LMF API query LSI module RDF answer Linked Media Store Linked Services Infrastructure New module To-beinterpreted answer (wrapped) query „lifting“ and „lowering“ handled by LSI Some online content source
    31. 31. Linked Services Infrastructure RDF response format based on W3C Media Ontology Types are e.g. Image, VideoTrack Locator points to the actual resource URL hasKeyword links the resource to LOD URIs related to the resource We can include further metadata as provided from the Web API (e.g. when uploaded, no of views, duration, size...)
    32. 32. Channel enrichment • ConnectME player with optional styling (customer branding) • Pre-selection of content sources and presentation (widgets) by administrator (administration interface) • On-the-fly video enrichment by selected widgets at play-out (user interface) • Administrator access to player analytics – playback, click-through, in-widget actions
    33. 33. Hypervideo playout & browsing Based on open source LIME player project HTML5 & CSS3 based Video format support as per browser support - MPEG-4 (IE, Safari, Android) - WebM and OGV (Firefox, Chrome, Android, Opera) Production player Development player Software project (Apache 2.0 license)
    34. 34. Player implementations • Desktop – Administrative & user interface – Widgets as extra webpage regions („standard“ player) – Fullscreen (with widgets as overlays) • TV – GoogleTV (fullscreen, no video pause, supports remote control) • Mobile – Tablet version (fullscreen adapted to touch, min 7“ screen) „The“ player incorporates both standard and fullscreen impl.
    35. 35. Player widget model Standard widget model is tiles-based* and abstracts the Interaction model from • Desktop: point and click • Mobile: touch • TV: remote control * A tile is 300x300px and a typical widget is max 600x600px
    36. 36. „Core“ player widgets • Info (DBPedia) • Media (LSI) • Map
    37. 37. „Extended“ player widgets Extended widget model needs definition of: - Annotation trigger (concept type) - Content to display (query CMF) - How to display it (tiling)
    38. 38. „Live TV“ player widgets PS Media use case defines these additional widgets - Actor - Character - Director
    39. 39. Demos Production workflow has demos of ConnectME, at IPTV example: Choose „Fahrenheit 9/11“ and TV player Mobile tourism example: Choose „Schladming“ and tablet player