Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
User Engagement –  A Scientific Challenge Mounia Lalmas Yahoo! Research Barcelona [email_address] Ioannis Arapakis Ricardo...
Outline <ul><li>Motivation and Definition </li></ul><ul><li>Characteristics of User Engagement </li></ul><ul><li>Current m...
Motivation <ul><li>User engagement is a quality of user experience that emphasizes the  positive  aspects of interaction –...
Would a user engage with this web site? http://www.nhm.ac.uk/
Would a user engage with this web site? http://www.amazingthings.org/  (art event calendar)
Would a user engage with this web site? http://www.lowpriceskates.com/   (e-commerce – skating)
Would a user engage with this web site? http://chiptune.com/  (music repository)
Would a user engage with this web site? http://www.theosbrinkagency.com/  (photographer)
What is user engagement? <ul><ul><ul><ul><li>“ The emotional, cognitive and/or behavioural connection that exists, at any ...
Characteristics of user engagement (I)
Characteristics of user engagement (II)
Forrester Research – The four I’s Measuring Engagement, Forrester Research, June 2008
Peterson etal  Engagement measure – 8 indices <ul><li>Click Depth Index:  page views </li></ul><ul><li>Duration Index :  t...
Subjective measures Record a user’s perception (generally self-reported) of the technology Two-part processes: Increased u...
Objective measures <ul><li>To overcome the subjectivity of post-experience questionnaires </li></ul><ul><li>To measure eng...
Objective measures –  Online activities   Proxy of user engagement <ul><li>bounce rates, CTR, days since last visit, exit ...
Characteristics and measures S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Pos...
Diagnostic and what we can do <ul><li>Diagnostic:  work exists, but fragmented. In particular: </li></ul><ul><ul><li>What ...
Page stylistics + layout  +links + saliency Measurements and methodologies + online analytics metrics (dwell time, …) + qu...
Focus – Two “opposite” studies models USER ENGAGEMENT 1. Models of user engagement 2. On saliency, attention and  positive...
1. Models of user engagement <ul><li>User engagement characteristics depend on the web application: </li></ul><ul><ul><li>...
Data and Metrics <ul><li>Interaction data, 2M users, July 2011, 73 US sites </li></ul>Popularity #Users Number of distinct...
Diversity in user engagement <ul><li>Users and Loyalty </li></ul><ul><li>Sites have different user groups </li></ul><ul><l...
Methodology General models User-based models Time-based models Dimensions 8 metrics 5 user groups 8 metrics per user group...
Models of user engagement <ul><li>6 general models </li></ul><ul><li>Popularity, activity and loyalty are  independent  fr...
Models of user engagement <ul><li>User-based [7 models] </li></ul><ul><li>Models based on engagement per  user group </li>...
Relationships between models <ul><li>Groups of models are independent from each other </li></ul><ul><li>Example:  </li></u...
Models of user engagement –  Recap & Next <ul><li>User engagement is complex and standard metrics capture only a part of i...
Focus – Two “opposite” studies models USER ENGAGEMENT 1. Models of user engagement 2. On saliency, attention and  positive...
2.  Saliency, attention and positive affect <ul><li>How the visual catchiness ( saliency ) of “relevant” information impac...
Manipulating saliency salient condition non-salient condition Web page screenshot Saliency maps
Study design <ul><li>8 tasks = finding latest news or headline on celebrity or entertainment topic </li></ul><ul><li>Affec...
PANAS  (10 positive items and 10 negative items) <ul><li>You feel this way right now, that is, at the present moment  </li...
7-item focused attention subscale  (part of the 31-item user engagement scale)   5-point scale (strong disagree to strong ...
Study protocol <ul><li>Mechanical Turk: Introduction/Consent, HIT acceptance </li></ul><ul><li>Link to external survey; en...
Saliency and positive affect <ul><li>When headlines are visually non-salient </li></ul><ul><ul><li>users are slow at findi...
Saliency  and  focused attention <ul><li>Adapted focused attention subscale from the online shopping domain to entertainme...
Saliency and user engagement –  Recap & Next <ul><li>interaction of saliency, focused attention, and affect, together with...
The big picture ……….… my vision Page stylistics + layout  +links + saliency Measurements and methodologies + online analyt...
Gracias <ul><ul><li>[email_address] </li></ul></ul><ul><ul><li>www.dcs.gla.ac.uk/~mounia </li></ul></ul>
Upcoming SlideShare
Loading in …5
×

User Engagement - A Scientific Challenge

4,165 views

Published on

Published in: Technology, Design
  • Be the first to comment

User Engagement - A Scientific Challenge

  1. 1. User Engagement – A Scientific Challenge Mounia Lalmas Yahoo! Research Barcelona [email_address] Ioannis Arapakis Ricardo Baeza-Yates Georges Dupret Janette Lehmann Lori McCay-Peet Vidhya Navalpakkam Elad Yom-Tov Collaborators
  2. 2. Outline <ul><li>Motivation and Definition </li></ul><ul><li>Characteristics of User Engagement </li></ul><ul><li>Current measurements </li></ul><ul><li>Vision and focus </li></ul><ul><li>Two “opposite” studies </li></ul><ul><ul><li>Models of user engagement </li></ul></ul><ul><ul><li>Saliency and user engagement </li></ul></ul>
  3. 3. Motivation <ul><li>User engagement is a quality of user experience that emphasizes the positive aspects of interaction – in particular the fact of being captivated by the technology. </li></ul><ul><li>Successful technologies are not just used, they are engaged with . </li></ul><ul><li>A web interface that is boring, a multimedia presentation that does not captivate users’ attention or an online forum that fails to engender a sense of community are quickly dismissed with a simple mouse click. </li></ul><ul><li>(O’Brian and Toms, 2008) </li></ul><ul><li>User engagement is how we nurture and build a community. </li></ul><ul><li>(John Byrne, Business Weeks’ online editor 2009) </li></ul>
  4. 4. Would a user engage with this web site? http://www.nhm.ac.uk/
  5. 5. Would a user engage with this web site? http://www.amazingthings.org/ (art event calendar)
  6. 6. Would a user engage with this web site? http://www.lowpriceskates.com/ (e-commerce – skating)
  7. 7. Would a user engage with this web site? http://chiptune.com/ (music repository)
  8. 8. Would a user engage with this web site? http://www.theosbrinkagency.com/ (photographer)
  9. 9. What is user engagement? <ul><ul><ul><ul><li>“ The emotional, cognitive and/or behavioural connection that exists, at any point in time and over time, between a user and a technological resource” </li></ul></ul></ul></ul><ul><ul><li>“ Newish” field of research stemming from the increasing interaction between users and web services and technologies </li></ul></ul><ul><ul><li>Current work focuses on two aspects </li></ul></ul><ul><ul><li>What are the characteristics of user engagement? </li></ul></ul><ul><ul><li>How can we measure user engagement? </li></ul></ul><ul><ul><ul><ul><li>Subjective measures </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Objective measures </li></ul></ul></ul></ul>
  10. 10. Characteristics of user engagement (I)
  11. 11. Characteristics of user engagement (II)
  12. 12. Forrester Research – The four I’s Measuring Engagement, Forrester Research, June 2008
  13. 13. Peterson etal Engagement measure – 8 indices <ul><li>Click Depth Index: page views </li></ul><ul><li>Duration Index : time spent </li></ul><ul><li>Recency Index: rate at which users return over time </li></ul><ul><li>Loyalty Index: level of long-term interaction the user has with the site or product (frequency) </li></ul><ul><li>Brand Index: apparent awareness of the user of the brand, site, or product (search terms) </li></ul><ul><li>Feedback Index: qualitative information including propensity to solicit additional information or supply direct feedback </li></ul><ul><li>Interaction Index: user interaction with site or product (click, upload, transaction) </li></ul>Peterson etal. Measuring the immeasurable: visitor engagement, WebAnalyticsDemystified, September 2008
  14. 14. Subjective measures Record a user’s perception (generally self-reported) of the technology Two-part processes: Increased use of crowd-sourcing based studies
  15. 15. Objective measures <ul><li>To overcome the subjectivity of post-experience questionnaires </li></ul><ul><li>To measure engagement over large populations and over time </li></ul>
  16. 16. Objective measures – Online activities Proxy of user engagement <ul><li>bounce rates, CTR, days since last visit, exit rate, entrance rate, frequency, number of new visitors, number of visitors, number of returning visitors, number of bookmarks (e.g. on Delicious), number of comments posted, number of content syndication (RSS), content contribution (adding a comment, adding a review, rating, uploading an image or video), number of repeated visitors, page views, time spent (dwell time), visits per visitors, number of tags (e.g. on Flickr), number of emailed and printed stories, number of Facebook likes, number of re-tweets, number of messages sent (instant, email), number of conversions (e.g. subscribing, buying), number of Facebook fan pages, number of search queries, common paths (e.g. from front page to mail tool then exit), external comments and reviews (for products), etc … </li></ul>
  17. 17. Characteristics and measures S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.
  18. 18. Diagnostic and what we can do <ul><li>Diagnostic: work exists, but fragmented. In particular: </li></ul><ul><ul><li>What and how to measure depend on services and goals </li></ul></ul><ul><ul><li>Lack of understanding of how to relate subjective and objective measures </li></ul></ul><ul><ul><li>What I (we) have done: </li></ul></ul><ul><ul><li>“ Towards” Models of user engagement </li></ul></ul><ul><ul><li>Stylistics and “engagement” </li></ul></ul><ul><ul><ul><li>Saliency, interest, attention and positive affect </li></ul></ul></ul><ul><ul><ul><li>Front page styles and downstream engagement </li></ul></ul></ul><ul><ul><ul><li>Automatic linking and reading experience </li></ul></ul></ul>
  19. 19. Page stylistics + layout +links + saliency Measurements and methodologies + online analytics metrics (dwell time, …) + questionnaires + crowd-sourcing - biometrics (eye tracking, mouse tracking, …) Goals + Models of user engagement - Metrics of user engagement Engagement types + attention + emotion + activity + popularity + loyalty - functionality +/- intent & interest user engagement within and across site
  20. 20. Focus – Two “opposite” studies models USER ENGAGEMENT 1. Models of user engagement 2. On saliency, attention and positive effect activity metrics loyalty metrics popularity metrics hobbies navigation social media e-commerce magazine sport news search weather mail weekly news … focused attention affect (emotion) saliency
  21. 21. 1. Models of user engagement <ul><li>User engagement characteristics depend on the web application: </li></ul><ul><ul><li>e.g. reading mail or browsing a news portal results in different types of engagement </li></ul></ul><ul><li>Diversity of engagement </li></ul><ul><li>Models of engagement through clustering using various criteria </li></ul><ul><ul><li>e.g. user types, temporal aspects </li></ul></ul>
  22. 22. Data and Metrics <ul><li>Interaction data, 2M users, July 2011, 73 US sites </li></ul>Popularity #Users Number of distinct users #Visits Number of visits #Clicks Number of clicks Activity ClickDepth Average number of page views per visit. DwellTimeA Average time per visit Loyalty ActiveDays Number of days a user visited the site ReturnRate Number of times a user visited the site DwellTimeL Average time a user spend on the site.
  23. 23. Diversity in user engagement <ul><li>Users and Loyalty </li></ul><ul><li>Sites have different user groups </li></ul><ul><li>Proportion of user groups is site-dependent </li></ul><ul><li>Time and Popularity </li></ul><ul><li>Site engagement can be periodic or contains peaks </li></ul>Engagement of a site depends on users and time mail, social media shopping, entertainment media (special events) daily activity, navigation media, entertainment
  24. 24. Methodology General models User-based models Time-based models Dimensions 8 metrics 5 user groups 8 metrics per user group weekdays, weekend 8 metrics per time span #Dimensions 8 40 16 Kernel k-means with Kendall tau rank correlation kernel Nb of clusters based on eigenvalue distribution of kernel matrix Significant metric values with Kruskal-Wallis/Bonferonni #Clusters (Models) 6 7 5 Analysing cluster centroids = models
  25. 25. Models of user engagement <ul><li>6 general models </li></ul><ul><li>Popularity, activity and loyalty are independent from each other </li></ul><ul><li>Popularity and loyalty are influenced by external and internal factors </li></ul><ul><ul><li>e.g. frequency of publishing new information, events, personal interests </li></ul></ul><ul><li>Activity depends on the structure of the site </li></ul>Models based on engagement metrics interest-specific e-commerce, configuration periodic media
  26. 26. Models of user engagement <ul><li>User-based [7 models] </li></ul><ul><li>Models based on engagement per user group </li></ul><ul><li>Time-based [5 models] </li></ul><ul><li>Models based on engagement over weekdays and weekend </li></ul>Models based on engagement metrics, user and time navigation game, sport hobbies, interest-specific daily news <ul><li>Sites of the same type (e.g. mainstream media) do not necessarily belong to the same model </li></ul><ul><li>The groups of models describe different aspects of engagement, i.e. they are independent from each other </li></ul>
  27. 27. Relationships between models <ul><li>Groups of models are independent from each other </li></ul><ul><li>Example: </li></ul><ul><ul><li>Model m u2 </li></ul></ul><ul><ul><ul><li>[high popularity and activity in all user groups, increasing loyalty] </li></ul></ul></ul><ul><ul><ul><li>50% to model m t2 [ high popularity on weekends and high loyalty on weekdays ] </li></ul></ul></ul><ul><ul><ul><li>50% to model m t3 [ high activity and loyalty on weekends ] </li></ul></ul></ul>Variance of Information [0,5.61] General User Time General 0.00 3.50 4.23 User 3.50 0.00 4.25 Time 4.23 4.25 0.00
  28. 28. Models of user engagement – Recap & Next <ul><li>User engagement is complex and standard metrics capture only a part of it </li></ul><ul><li>User engagement depends on users and time </li></ul><ul><li>First step towards a taxonomy of models of user engagement … and associated metrics </li></ul><ul><li>Next </li></ul><ul><ul><li>Interaction between models </li></ul></ul><ul><ul><li>Interaction between sites (complex networks) </li></ul></ul><ul><ul><li>User demographics, time of the day, geo-location, etc </li></ul></ul>
  29. 29. Focus – Two “opposite” studies models USER ENGAGEMENT 1. Models of user engagement 2. On saliency, attention and positive effect activity metrics loyalty metrics popularity metrics hobbies navigation social media e-commerce magazine sport news search weather mail weekly news … focused attention affect (emotion) saliency
  30. 30. 2. Saliency, attention and positive affect <ul><li>How the visual catchiness ( saliency ) of “relevant” information impacts user engagement metrics such as focused attention and emotion (affect) </li></ul><ul><ul><li>focused attention refers to the exclusion of other things </li></ul></ul><ul><ul><li>affect relates to the emotions experienced during the interaction </li></ul></ul><ul><li>Saliency model of visual attention developed by Itti and Koch </li></ul><ul><li>L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 10-12 (2000). </li></ul>
  31. 31. Manipulating saliency salient condition non-salient condition Web page screenshot Saliency maps
  32. 32. Study design <ul><li>8 tasks = finding latest news or headline on celebrity or entertainment topic </li></ul><ul><li>Affect measured pre- and post- task using the Positive e.g. “determined”, “attentive” and Negative e.g. “hostile”, “afraid ” Affect Schedule (PANAS) </li></ul><ul><li>Focused attention measured with 7-item focused attention subscale e.g. “ I was so involved in my news tasks that I lost track of time”, “ I blocked things out around me when I was completing the news tasks ” and perceived time </li></ul><ul><li>Interest level in topics (pre-task) and questionnaire (post-task) e.g. “I was interested in the content of the web pages”, “I wanted to find out more about the topics that I encountered on the web pages” </li></ul><ul><li>189 (90+99) participants from Amazon Mechanical Turk </li></ul>
  33. 33. PANAS (10 positive items and 10 negative items) <ul><li>You feel this way right now, that is, at the present moment </li></ul><ul><li>[1 = very slightly or not at all; 2 = a little; 3 = moderately; </li></ul><ul><li>4 = quite a bit; 5 = extremely] </li></ul><ul><li>[randomize items] </li></ul><ul><ul><li>distressed, upset, guilty, scared, hostile, irritable, ashamed, nervous, jittery, afraid </li></ul></ul><ul><ul><li>interested, excited, strong, enthusiastic, proud, alert, inspired, determined, attentive, active </li></ul></ul>D. Watson, L.A. Clark & A. Tellegen. Development and validation of brief measures of positive and negative affect: The PANAS Scales. Journal of Personality and Social Psychology, 47 (1988).
  34. 34. 7-item focused attention subscale (part of the 31-item user engagement scale) 5-point scale (strong disagree to strong agree) <ul><li>I lost myself in this news tasks experience </li></ul><ul><li>I was so involved in my news tasks that I lost track of time </li></ul><ul><li>I blocked things out around me when I was completing the news tasks </li></ul><ul><li>When I was performing these news tasks, I lost track of the world around me </li></ul><ul><li>The time I spent performing these news tasks just slipped away </li></ul><ul><li>I was absorbed in my news tasks </li></ul><ul><li>During the news tasks experience I let myself go </li></ul>H.L. O'Brien. Defining and Measuring Engagement in User Experiences with Technology. PhD Thesis, 2008.
  35. 35. Study protocol <ul><li>Mechanical Turk: Introduction/Consent, HIT acceptance </li></ul><ul><li>Link to external survey; entry of MTurk worker ID </li></ul><ul><li>Pre-task PANAS </li></ul><ul><li>Pre-task topic interest questions </li></ul><ul><li>8 Non-Salient tasks 8 Salient tasks </li></ul><ul><li>Self-report of time spent finding headlines </li></ul><ul><li>Post-task PANAS </li></ul><ul><li>Self-report of affect </li></ul><ul><li>Focused attention scale </li></ul><ul><li>Self-report of focused attention </li></ul><ul><li>Interest and task ease questions </li></ul><ul><li>Demographics questionnaire </li></ul><ul><li>Comments </li></ul><ul><li>Return to MTurk to submit HIT </li></ul>
  36. 36. Saliency and positive affect <ul><li>When headlines are visually non-salient </li></ul><ul><ul><li>users are slow at finding them, report more distraction due to web page features, and show a drop in affect </li></ul></ul><ul><li>When headlines are visually catchy or salient </li></ul><ul><ul><li>user find them faster, report that it is easy to focus, and maintain positive affect </li></ul></ul><ul><li>Saliency is helpful in task performance, focusing/avoiding distraction and in maintaining positive affect </li></ul>
  37. 37. Saliency and focused attention <ul><li>Adapted focused attention subscale from the online shopping domain to entertainment news domain </li></ul><ul><li>Users reported “easier to focus in the salient condition” BUT no significant improvement in the focused attention subscale or differences in perceived time spent on tasks </li></ul><ul><li>User interest in web page content is a good predictor of focused attention, which in turn is a good predictor of positive affect </li></ul>
  38. 38. Saliency and user engagement – Recap & Next <ul><li>interaction of saliency, focused attention, and affect, together with user interest , is complex </li></ul><ul><li>Next: </li></ul><ul><ul><li>include web page content as a quality of user engagement in focused attention scale </li></ul></ul><ul><ul><li>more “realistic” user (interactive) reading experience </li></ul></ul><ul><ul><li>bio-metrics (mouse-tracking, eye-tracking, facial expression, etc) </li></ul></ul>L. McCay-Peet, M. Lalmas, V. Navalpakkam. On saliency, affecr and focused attention, CHI 2012
  39. 39. The big picture ……….… my vision Page stylistics + layout +links + saliency Measurements and methodologies + online analytics metrics (dwell time, …) + questionnaires + crowd-sourcing - biometrics (eye tracking, mouse tracking, …) Goals + Models of user engagement - Metrics of user engagement Engagement types + attention + emotion + activity + popularity + loyalty - functionality +/- intent & interest user engagement within and across site
  40. 40. Gracias <ul><ul><li>[email_address] </li></ul></ul><ul><ul><li>www.dcs.gla.ac.uk/~mounia </li></ul></ul>

×