• Save
Contextualised user profiling in networked media environments
Upcoming SlideShare
Loading in...5
×
 

Contextualised user profiling in networked media environments

on

  • 625 views

The talk was delivered by Dorothea Tsatsou at the workshop at the UMAP 2012 conference, 16 - 20 July 2012, Montreal, Canada. More info: http://bit.ly/Qvnbz4

The talk was delivered by Dorothea Tsatsou at the workshop at the UMAP 2012 conference, 16 - 20 July 2012, Montreal, Canada. More info: http://bit.ly/Qvnbz4

Statistics

Views

Total Views
625
Views on SlideShare
583
Embed Views
42

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 42

http://www.linkedtv.eu 42

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Annotating audiovisual content with concepts Using that annotation to (semi-)automatically link parts of audiovisual content to Web content Providing an interactive video experience for the user to browse objects within the video program
  • Annotating audiovisual content with concepts Using that annotation to (semi-)automatically link parts of audiovisual content to Web content Providing an interactive video experience for the user to browse objects within the video program
  • Annotating audiovisual content with concepts Using that annotation to (semi-)automatically link parts of audiovisual content to Web content Providing an interactive video experience for the user to browse objects within the video program
  • Annotating audiovisual content with concepts Using that annotation to (semi-)automatically link parts of audiovisual content to Web content Providing an interactive video experience for the user to browse objects within the video program
  • Location: determine the environment close to the TV. During time, the system will be able to learn the in/out regions but also where people have a high probability to focus on the TV (sofa) or to talk together (dinner table). Identity and context: a person can be recognized. This point is also important to know the number of people, if they are already known or not, and to extract biometric features mainly about their age and gender. Orientation: detect if the focus of attention is on the TV or not based on the direction of the body of the user. Distance and static features: the distance of the users relative to the TV can be used to activate implicit or explicit interaction or to understand the relations between the different users and the TV (who is really interested, who is just there to talk to the others…). Motion and dynamic features: changes of distance and orientation over time are interesting to analyze the evolution of the interest of people in the content delivered by the TV.

Contextualised user profiling in networked media environments Contextualised user profiling in networked media environments Presentation Transcript

  • Television Linked To The WebDorothea Tsatsou1, Lyndon Nixon2, Matei Mancas3, Miroslav Vacura4, Rüdiger Klein5, Julien Leroy3, Jaroslav Kuchař4, Tomáš Kliegr4, Manuel Kober5, Maria Loli1, Vasileios Mezaris1 Contextualised user profiling in networked media environments1 CERTH-ITI, Thessaloniki, Greece 4 University of Economics Prague, Prague, Czech Republic2 STI International, Vienna, Austria 5 Fraunhofer IAIS, Sankt Augustin, Bonn, Germany3 University of Mons, Mons, Belgium 2nd Augmented User Modeling workshop, UMAP 2012, Montreal, July 2012 www.linkedtv.eu
  • A new era to TV viewing www.linkedtv.eu 60% of Americans engage in couch potato multitasking 86% of TV viewers with broadband surf while they watch Social TV 38% of networked media users discussing what theyre watching on social media (53% in the 16 to 23- year-old demographic)-- Nielsen (2009). Three screen report. Technical report, Nielsen Company-- Yahoo! and Nielsen (2010). Mobile shopping framework - the role of mobile devices inthe shopping process. Technical report, Yahoo! and The Nielsen Company http://www.designbynotion.com/metamirror-next-generation-tv-- Ovum survey, published in TheRegister, 6 October 2011 Information Technologies Institute 2 Centre for Research and Technology Hellas
  • Towards networked media… www.linkedtv.euSecond Screen content push – breakthrough with HTML5 mobile Second screen apps show related content without disturbing the TV view The rise of Smart TVs LG SmartTV, pic courtesy http://www.wired.com/gadgetlab/2011/01/lg-smart-tv/ Information Technologies Institute 3 Centre for Research and Technology Hellas
  • …to LinkedTV www.linkedtv.eu TV and Web access are unified on the TV device but not the experience Interweaving TV and Web content into a single experience a focus of the LinkedTV project Information Technologies Institute4 Centre for Research and Technology Hellas
  • LinkedTV ― Television Linked To the Web www.linkedtv.euVision: 12 Excellent Partners ubiquitously online cloud of Fraunhofer Eurecom Networked Audio-Visual Content STI GMBH Condat decoupled from place, device or CERTH BEELD EN GELUID source UEP Noterik UMONS U. ST GALLEN Aim: CWI RBB provide interactive multimedia service for non-professional end-users focus television broadcast content as seed videos Web: http://www.linkedtv.eu Information Technologies Institute 5 Centre for Research and Technology Hellas
  • LinkedTV Workflow www.linkedtv.eu Overall Architecture Use Case Scenarios Intelligent Video Analysis Linking Hypervideo to Web Content Contextualization and Personalization Interface and Presentation Engine Information Technologies Institute6 Centre for Research and Technology Hellas
  • LinkedTV Workflow www.linkedtv.eu Interactive News Show Overall Architecture Hyperlinked Documentary Use Case Scenarios Media Arts Intelligent Video Analysis Linking Hypervideo to Web Content Contextualization and Personalization Interface and Presentation Engine Information Technologies Institute7 Centre for Research and Technology Hellas
  • LinkedTV Workflow www.linkedtv.eu Overall Architecture Use Case Scenarios Intelligent Video Analysis Cubism Linking Hypervideo to Web Content Contextualization Fauvism and Personalization Interface and Presentation Engine Expressionism Information Technologies Institute8 Centre for Research and Technology Hellas
  • LinkedTV Workflow www.linkedtv.eu Overall Architecture Cubism Use Case Scenarios Fauvism Expressionism Intelligent Video Analysis Linking Hypervideo to Web Content Contextualization and Personalization Interface and Presentation Engine CONTENT ENRICHMENT Information Technologies Institute9 Centre for Research and Technology Hellas
  • LinkedTV Workflow www.linkedtv.eu Overall Architecture Use Case Scenarios Intelligent Video Analysis Linking Hypervideo to Web Content Contextualization and Personalization Interface and Presentation Engine Information Technologies Institute10 Centre for Research and Technology Hellas
  • Challenge www.linkedtv.eu Digital information overload Digital information heterogeneity Information Technologies Institute11 Centre for Research and Technology Hellas
  • Transactional user modeling – Information sources www.linkedtv.euExplicit user preferences Demographics  Strereotypes Direct definition of pre-determined concepts/categories/keywordsContent (media, external web resources) consumption history Data/text mining Ratings Actions on player/browserSocial interaction Comments, likes Peer-to-peer similaritiesSemantic knowledge bases (ontologies) Information Technologies Institute 12 Centre for Research and Technology Hellas
  • Information sources pros & cons www.linkedtv.euExplicit user preferences Social interaction + Accurate + Intuitive (unforeseen preferences) - Outdated, intrusive - Cold start, scalability, data sparcityContent consumption history Semantic knowledge bases (ontologies) + Straightforward, indicative + Information from the get-go - Video, audio, articles, wikis, tags, rss + Uniform, finite vocabulary feeds…: too diverse and non- - Manual creation, lack of mappings uniformly characterized - Cold start, scalability, data sparcity Solution: Hybrid Information Technologies Institute 13 Centre for Research and Technology Hellas
  • Context indicators based on transactions www.linkedtv.euTime Time of day, season etcLocation At home, at work, out of the country etcActions on the player/browser Play, stop, rewind, skip, bookmark, scrollRecent content consumptions Goal: Recognize persistent preference patterns for certain contexts Information Technologies Institute 14 Centre for Research and Technology Hellas
  • Learning from transactions www.linkedtv.euNature of transaction Positive, negativePreference impact Weighted preferencesUser behaviour pattern discovery Association rules Clustering Utility functions Information Technologies Institute 15 Centre for Research and Technology Hellas
  • View the viewers www.linkedtv.eu TV with integrated cameras are on the way (Samsung, Phillips, ...) Being viewed is well accepted at homes  Kinect  Xbox  SoftKinetic Orange TV Information Technologies Institute 16 Centre for Research and Technology Hellas
  • Features that can be extracted www.linkedtv.euLocation: determine the environment close to the TV.Identity and (physical) context: a person can be recognized.Orientation: gaze-based focus of attentionDistance and static features: the distance of the users relative to the TVMotion and dynamic features: behavioural changes Information Technologies Institute 17 Centre for Research and Technology Hellas
  • Feature extraction www.linkedtv.euLocation-based features depending on the viewer position, different interfaces are displayed Greenberg, S., Marquardt, N., et al. Proxemic interactions:the new ubicomp? interactions 18, ACM (2011), 42–50. Information Technologies Institute 18 Centre for Research and Technology Hellas
  • Feature extraction www.linkedtv.euOrientation (left), identity and context (right) Information Technologies Institute 19 Centre for Research and Technology Hellas
  • Feature extraction www.linkedtv.euDistance features (with TV and other people) Information Technologies Institute 20 Centre for Research and Technology Hellas
  • Feature extraction www.linkedtv.euDynamic features (sudden changes in position, orientation ...) Information Technologies Institute 21 Centre for Research and Technology Hellas
  • Holistic user modeling www.linkedtv.euWhat information do we care about? All: transactional, behavioural, social The more you know about the user, the better: implicit dataHow to unify diverse implicit information? One uniform vocabulary and conceptualization about the world Lightweight, finite, expressive knowledgeSolution: Semantic user modeling Information Technologies Institute 22 Centre for Research and Technology Hellas
  • Semantic User Profiling www.linkedtv.eu Explicit User Information User-defined preferences Stereotypes Demographics User User Profile:Requirements Update Structure Implicit User Information & Formalize Weighted concepts Extract Learn Understand (preferences), quantified by ontology properties and relations between them (conjunction, disjunction, negation) Tracking Ontologies Information Technologies Institute 23 Centre for Research and Technology Hellas
  • Requirements www.linkedtv.euOntology: formal Knowledge base (not only the vocabulary) Describe the “world” Reflect the user E.g. actorX playsIn movieY, pollution isRelatedTo environment, anchorman is-a person, attention(low) → disinterest etc Compromise between coverage and expressivitySemantic content and actions classification Map raw data to ontologyExpressive representation schema Suitable for logical inferencing Information Technologies Institute 24 Centre for Research and Technology Hellas
  • Understanding content www.linkedtv.eu Multimedia content External content Direct mappings Textual analysis for mappings discovery (classification) (GATE, SProUT, OpenCalais) Linked Social Web Activities Media Layer Ontology Common vocabulary Classified consumption behaviour {c∙w, a:c∙w, …..} Information Technologies Institute25 Centre for Research and Technology Hellas
  • Understanding the relation of the content to the user www.linkedtv.euInterests vs Disinterests Click behaviour (actions on player, dwell time) Physical reactionWeight: impact of the concept to the user Based just on the importance of a concept in the annotated/classified content (scene) User engagement Click behaviour (actions on player, dwell time) Physical state recognition (attention, mood) Social interaction (likes, shares etc)Profile concept weights updated: Upon every transaction (content consumption), based on frequency Over time (instantly every minute): Time decay Information Technologies Institute 26 Centre for Research and Technology Hellas
  • Structure and advantages of the profile www.linkedtv.eu Profile ← (¬ ConceptA(X) ∙ w1 ∧ ConceptB(a) ∙ w2) ∨ ∃relationA.ConceptC(b) ∙ w3Lightweight & uniform user model Storage even in limited resource devices, scalabilityAble to represent and take advantage of more complex knowledge Relations of interests/knowledge concepts Constructors and rules (conjunction, disjunction, disjointness) Richer semantics (inverse, transitive properties, complements) Especially useful to represent/discern disinterestsCan easily be used with reduced semantics for simple inferencing methods (spreading activation, clustering etc)Easily breakable to contextual instances Information Technologies Institute 27 Centre for Research and Technology Hellas
  • Contextualization Workflow www.linkedtv.eu Information Technologies Institute28 Centre for Research and Technology Hellas
  • The LinkedTV personalisation & contextualisationapproach www.linkedtv.eu Extract, understand & structure in a machine-understandable way user preferences in regard to contextUnderstand what the data means (to the world & to the user)Augment preferences with additional related informationUser information manifested in a machine-understandable formatLearn impact and relative patterns of preferencesHarvest mass intelligenceDetermine the reactional and physical state of the userUnderstand and determine contextual variations of the user profile Provide a conceptual user profile able to be used for semantic inferencing Information Technologies Institute 29 Centre for Research and Technology Hellas
  • One step further www.linkedtv.euKnowledge pulling Alleviate information overload in the inferencing stage by reducing the knowledge based on user contextOntology learning Determine new or group-specific knowledgePrivacy preservation Minimize client-server communication Anonymization & encryption techniques Information Technologies Institute 30 Centre for Research and Technology Hellas
  • www.linkedtv.eu Questions ?More information:dorothea@iti.grwww.linkedtv.eu@linkedtv Information Technologies Institute31 Centre for Research and Technology Hellas