Characteristics elaborate the notion of engagement over 3 broad dimensions: emotional, cognitive and behavioural. We identified 8 of them based on previous works / suggested by us. Attention -> exclusivity Affect -> lack of fun can lead as a barrier to shopping (O’Brian) Aesthetics -> promotes attention, stimulate curiosity Endurability -> does the experience meet the expectation?
Novelty -> freshness of content / has to be carefully balanced Richness = complexity of thoughts and actions, increased complexity/ Affect = relates to the emotion experienced during the interaction (fun and enjoyment)
Session, focus and engagement
Session, focus and engagement Mounia Lalmas Yahoo! Research Barcelona firstname.lastname@example.org
A bit about myself 1999-2008: Lecturer (assistant professor) to Professor at Queen Mary, University of London 2008-2010 Microsoft Research/RAEng Research Professor at the University of Glasgow (and lived outside London) 2011- Visiting Principal Scientist at Yahoo! Research Barcelona Research topics XML retrieval and evaluation (INEX) Quantum theory to model interactive information retrieval Aggregated search Bridging the digital divide Models and measures of user engagement
Message and Outline Interaction and search Beyond result relevance Beyond search session Towards “engagement” 1. Motivations 2. Engagement 3. Future directions
Information Retrieval Over Query Sessions Retrieval Models & Ranking: How to analyze/model/predict user interactions and use these findings to improve retrieval performance? How can we adapt ranking/retrieval models and IR theory in the light of a sequence of user interactions. Evaluation & Test Collections: How can we evaluate retrieval system performance over entire query sessions? How can we build reusable test collections to study this IR task? How can we model/simulate user interactions over a session? User Interaction & Interfaces: How can we model user interactions so we can predict and improve the user experience over sessions? How can we design and perform user studies that reveal new information about users? How can we make use of implicit feedback from users?
Relevance in multimedia search Multimedia search activities often driven by entertainment needs, not byM. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011 information needs
Relevance in focused retrieval Relevance in context Table of Content Focused retrieval is about putting results (element, fact, passage) in context, to understand and trust themCourtesy Jaap Kamps, Zoltan Szlavik, Norbert Goevert
Courtesy of Janette LehmannBeyond search session On month browsing data, sample of sites (INT=Yahoo site, EXT=non Yahoo site) On month browsing data, sample of Yahoo! sitesusers spend more and more of their online session multi-tasking, e.g. emailing,reading news, searching for information ONLINE MULTI-TASKING navigating between sites, using browser tabs, etc seamless integration of social networks platforms into many services
Interactive IR … P Ingwersen, Human Aspects in IR, ESSIR 2011.
User Engagement – connecting three sides User engagement is a quality of user experience that emphasizes the positive aspects of interaction – in particular the fact of being captivated by the technology. Successful technologies are not just used, they are engaged with.user feelings: happy, sad, user mental states: concentrated, user interactions: click, readexcited, bored, … challenged, lost, interested … comment, recommend, buy, … The emotional, cognitive and/or behavioural connection that exists, at any point in time and over time, between a user and a technological resource
Characteristics of user engagement (I)S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),WSDM Workshop on User Modelling for Web Applications, 2011.
Characteristics of user engagement (II)S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),WSDM Workshop on User Modelling for Web Applications, 2011.
The four I’s Measuring Engagement, Forrester Research, June 2008
Objective measures – Online activitiesProxy of user engagement
Models of user engagement Online sites differ concerning their engagement! Games Search Users spend Users come much time per frequently and visit do not stay long Social media Special Users come Users come on frequently and average once per stay long time considered Service News Users visit site, Users come when needed periodically Is it possible to model these differences?
Data and MetricsInteraction data, 2M users, July 2011, 80 US sites Popularity #Users Number of distinct users #Visits Number of visits #Clicks Number of clicks Activity ClickDepth Average number of page views per visit. DwellTimeA Average time per visit Loyalty ActiveDays Number of days a user visited the site ReturnRate Number of times a user visited the site DwellTimeL Average time a user spend on the site.
Diversity in user engagement Engagement of a site depends on users and timeUsers and Loyalty Time and Popularity Sites have different user groups Site engagement can be periodic Proportion of user groups is site- or contains peaks dependent mail, social media media (special events) media, entertainment daily activity, shopping, navigation entertainment
Methodology General models User-based models Time-based modelsDimensions 5 user groups weekdays, weekend 8 metrics 8 metrics per user 8 metrics per time span group#Dimensions 8 40 16 Kernel k-means with Kendall tau rank correlation kernel Nb of clusters based on eigenvalue distribution of kernel matrix Significant metric values with Kruskal-Wallis/Bonferonni#Clusters(Models) 6 7 5 Analysing cluster centroids = models
Models of user engagement Models based on engagement metrics• 6 general models• Popularity, activity and loyalty are independent from each other• Popularity and loyalty are influenced by external and internal factors e.g. frequency of publishing new information, events, personal interests• Activity depends on the structure of the site interest- specific periodic media e-commerce, configuration
Models of user engagement Models based on engagement metrics, user and time User-based [7 models] Time-based [5 models] Models based on engagement per Models based on engagement user group over weekdays and weekendnavigation game, sport hobbies, daily news interest-specific Sites of the same type (e.g. mainstream media) do not necessarily belong to the same model The groups of models describe different aspects of engagement, i.e. they are independent from each other
Recap & Next User engagement is complex and standard metrics capture only a part of it First step towards a taxonomy of models of user engagement … and associated metrics Next Interaction between models Interaction between sites (multi-tasking) User demographics, time of the day, geo-location, etcJ. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.
3. Outline 1. Motivations • Engagement • Future directions • The three sides of user engagement • Interactive IR • Towards engagement
Let us revisit … connecting three sides + layout +links The three sides + saliency + content + emotional + cognitive + behavioral user engagement within and across site Measurements and methodologies + online analytics metrics (dwell time, CTR, …) + complex networks metricsGoals + Models of user engagement + questionnaires, surveys, … + Metrics of user engagement + crowd-sourcing + biometrics (eye tracking, mouse tracking, …)
Let us revisit … Interactive IR P Ingwersen, Human Aspects in IR, ESSIR 2011.
session, interaction, multi-tasking,network, search, relevance, … •I Aapakis, K Athanasakos, J Jose, A comparison of general vs personalised affective models for the prediction of topical relevance, SIGIR 2010. •J Huang, R White, S Dumais, No clicks, no problem: using cursor movements to understand and improve search, CHI 2011. • P Ingwersen & K Järvelin, The turn: integration of information seeking and retrieval in context, 2005. TOWARDS ENGAGEMENT
Information Retrieval Over Query Sessions Retrieval Models & Ranking: How to analyze/model/predict user T e EN nc interactions and use these findings to improve retrieval EM eva performance? How can we adapt ranking/retrieval models and IR theory in the light of a sequence of user interactions. AG rel E aNG nd Evaluation & Test Collections: How can we evaluate retrieval system performance over S D ion entire query sessions? How can we build R s over a session? reusable test collections to study this IR task? How can we A es model/simulate user interactions W s TO nd o User Interaction & Interfaces: How can we model user ywe can predict and improve the user experience over interactions e b so sessions? How can we design and perform user studies that reveal new information about users? How can we make use of implicit feedback from users?
Thank you email@example.com T e EN nc EM eva www.dcs.gla.ac.uk/~mounia AG rel NG nd E a D S ion R s A es W s TO nd yo be