Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Session, focus and engagement

1,416 views

Published on

ECIR 2012 workshop "Information Retrieval Over Query Sessions" Keynote

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Session, focus and engagement

  1. 1. Session, focus and engagement Mounia Lalmas Yahoo! Research Barcelona mounia@acm.org
  2. 2. A bit about myself 1999-2008: Lecturer (assistant professor) to Professor at Queen Mary, University of London 2008-2010 Microsoft Research/RAEng Research Professor at the University of Glasgow (and lived outside London) 2011- Visiting Principal Scientist at Yahoo! Research Barcelona Research topics  XML retrieval and evaluation (INEX)  Quantum theory to model interactive information retrieval  Aggregated search  Bridging the digital divide  Models and measures of user engagement
  3. 3. Message and Outline Interaction and search  Beyond result relevance  Beyond search session Towards “engagement” 1. Motivations 2. Engagement 3. Future directions
  4. 4. 1. Outline 1. Motivations • Relevance in multimedia search • Relevance in focused retrieval • Online multi-tasking • Engagement • Future directions
  5. 5. Information Retrieval Over Query Sessions  Retrieval Models & Ranking: How to analyze/model/predict user interactions and use these findings to improve retrieval performance? How can we adapt ranking/retrieval models and IR theory in the light of a sequence of user interactions.  Evaluation & Test Collections: How can we evaluate retrieval system performance over entire query sessions? How can we build reusable test collections to study this IR task? How can we model/simulate user interactions over a session?  User Interaction & Interfaces: How can we model user interactions so we can predict and improve the user experience over sessions? How can we design and perform user studies that reveal new information about users? How can we make use of implicit feedback from users?
  6. 6. Relevance in multimedia search Multimedia search activities often driven by entertainment needs, not byM. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011 information needs
  7. 7. Relevance in focused retrieval Relevance in context Table of Content Focused retrieval is about putting results (element, fact, passage) in context, to understand and trust themCourtesy Jaap Kamps, Zoltan Szlavik, Norbert Goevert
  8. 8. Courtesy of Janette LehmannBeyond search session On month browsing data, sample of sites (INT=Yahoo site, EXT=non Yahoo site) On month browsing data, sample of Yahoo! sitesusers spend more and more of their online session multi-tasking, e.g. emailing,reading news, searching for information  ONLINE MULTI-TASKING navigating between sites, using browser tabs, etc seamless integration of social networks platforms into many services
  9. 9. Interactive IR … P Ingwersen, Human Aspects in IR, ESSIR 2011.
  10. 10. 2. Outline 1. Motivations • (User) Engagement • Definition • Characteristics • Measuring • Models • Future directions
  11. 11. User Engagement – connecting three sides  User engagement is a quality of user experience that emphasizes the positive aspects of interaction – in particular the fact of being captivated by the technology.  Successful technologies are not just used, they are engaged with.user feelings: happy, sad, user mental states: concentrated, user interactions: click, readexcited, bored, … challenged, lost, interested … comment, recommend, buy, … The emotional, cognitive and/or behavioural connection that exists, at any point in time and over time, between a user and a technological resource
  12. 12. Characteristics of user engagement (I)S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),WSDM Workshop on User Modelling for Web Applications, 2011.
  13. 13. Characteristics of user engagement (II)S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),WSDM Workshop on User Modelling for Web Applications, 2011.
  14. 14. The four I’s Measuring Engagement, Forrester Research, June 2008
  15. 15. Measuring user engagement
  16. 16. Objective measures – Online activitiesProxy of user engagement
  17. 17. Models of user engagement Online sites differ concerning their engagement! Games Search Users spend Users come much time per frequently and visit do not stay long Social media Special Users come Users come on frequently and average once per stay long time considered Service News Users visit site, Users come when needed periodically Is it possible to model these differences?
  18. 18. Data and MetricsInteraction data, 2M users, July 2011, 80 US sites Popularity #Users Number of distinct users #Visits Number of visits #Clicks Number of clicks Activity ClickDepth Average number of page views per visit. DwellTimeA Average time per visit Loyalty ActiveDays Number of days a user visited the site ReturnRate Number of times a user visited the site DwellTimeL Average time a user spend on the site.
  19. 19. Diversity in user engagement Engagement of a site depends on users and timeUsers and Loyalty Time and Popularity Sites have different user groups  Site engagement can be periodic Proportion of user groups is site- or contains peaks dependent mail, social media media (special events) media, entertainment daily activity, shopping, navigation entertainment
  20. 20. Methodology General models User-based models Time-based modelsDimensions 5 user groups weekdays, weekend 8 metrics 8 metrics per user 8 metrics per time span group#Dimensions 8 40 16 Kernel k-means with Kendall tau rank correlation kernel Nb of clusters based on eigenvalue distribution of kernel matrix Significant metric values with Kruskal-Wallis/Bonferonni#Clusters(Models) 6 7 5 Analysing cluster centroids = models
  21. 21. Models of user engagement Models based on engagement metrics• 6 general models• Popularity, activity and loyalty are independent from each other• Popularity and loyalty are influenced by external and internal factors  e.g. frequency of publishing new information, events, personal interests• Activity depends on the structure of the site interest- specific periodic media e-commerce, configuration
  22. 22. Models of user engagement Models based on engagement metrics, user and time User-based [7 models] Time-based [5 models]  Models based on engagement per  Models based on engagement user group over weekdays and weekendnavigation game, sport hobbies, daily news interest-specific  Sites of the same type (e.g. mainstream media) do not necessarily belong to the same model  The groups of models describe different aspects of engagement, i.e. they are independent from each other
  23. 23. Recap & Next  User engagement is complex and standard metrics capture only a part of it  First step towards a taxonomy of models of user engagement … and associated metrics  Next  Interaction between models  Interaction between sites (multi-tasking)  User demographics, time of the day, geo-location, etcJ. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.
  24. 24. 3. Outline 1. Motivations • Engagement • Future directions • The three sides of user engagement • Interactive IR • Towards engagement
  25. 25. Let us revisit … connecting three sides + layout +links The three sides + saliency + content + emotional + cognitive + behavioral user engagement within and across site Measurements and methodologies + online analytics metrics (dwell time, CTR, …) + complex networks metricsGoals + Models of user engagement + questionnaires, surveys, … + Metrics of user engagement + crowd-sourcing + biometrics (eye tracking, mouse tracking, …)
  26. 26. Let us revisit … Interactive IR P Ingwersen, Human Aspects in IR, ESSIR 2011.
  27. 27. session, interaction, multi-tasking,network, search, relevance, … •I Aapakis, K Athanasakos, J Jose, A comparison of general vs personalised affective models for the prediction of topical relevance, SIGIR 2010. •J Huang, R White, S Dumais, No clicks, no problem: using cursor movements to understand and improve search, CHI 2011. • P Ingwersen & K Järvelin, The turn: integration of information seeking and retrieval in context, 2005. TOWARDS ENGAGEMENT
  28. 28. Information Retrieval Over Query Sessions  Retrieval Models & Ranking: How to analyze/model/predict user T e EN nc interactions and use these findings to improve retrieval EM eva performance? How can we adapt ranking/retrieval models and IR theory in the light of a sequence of user interactions. AG rel  E aNG nd Evaluation & Test Collections: How can we evaluate retrieval system performance over S D ion entire query sessions? How can we build R s over a session? reusable test collections to study this IR task? How can we A es model/simulate user interactions W s TO nd  o User Interaction & Interfaces: How can we model user ywe can predict and improve the user experience over interactions e b so sessions? How can we design and perform user studies that reveal new information about users? How can we make use of implicit feedback from users?
  29. 29. Thank you mounia@acm.org T e EN nc EM eva www.dcs.gla.ac.uk/~mounia AG rel NG nd E a D S ion R s A es W s TO nd yo be

×