Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
User Engagement:from Sites to a Network of Sites                or   The Network Effect Matters!           Ricardo Baeza-Y...
Outlineo  Motivation, definition and scopeo  Models of user engagemento  Networked user engagement                        ...
Motivation, Definition and Scopeo  Definition and scopeo  Characteristics of user engagemento  Measures of user engagement...
User Engagement – connecting three sides  Quality of the user experience that emphasizes the positive aspects of     the i...
Would a user engage with this web site?              http://www.nhm.ac.uk/       -5-
Would a user engage with this web site?      http://www.amazingthings.org/ (art event calendar)                           ...
Would a user engage with this web site?      http://www.lowpriceskates.com/ (e-commerce – skating)                        ...
Would a user engage with this web site?               http://chiptune.com/ (music repository)   -8-
Would a user engage with this web site?        http://www.theosbrinkagency.com/ (photographer)                            ...
Characteristics of user engagement (I)                      •  Users must be focused to be engaged  Focused attention   • ...
Characteristics of user engagement (II)                          •  Novelty, surprise, unfamiliarity and unexpected       ...
Forrester Research – The four I’s                •  Presence of a user  Involvement   •  Measured by e.g. number of visito...
Peterson et al Engagement measure - 8 indices      Click Depth Index: page views      Duration Index: time spent      Rece...
Measuring user engagement                  Measures	                            Characteristics	   Self-reported    Questi...
Interaction engagement – Online metricsProxy of user engagement                                          - 15 -
Diagnostic and what we can doDiagnostic: work exists, but fragmented.In particular: o  What and how to measure depend on s...
Models of User Engagement     Online sites differ concerning their engagement!             Games                         S...
Data and MetricsInteraction data, 2M users, July 2011, 80 USA sites   Popularity   #Users       Number of distinct users  ...
Diversity in User Engagement  Engagement of a site depends on users and timeUsers and Loyalty                     Time and...
Methodology              General models User-based models         Time-based modelsDimensions                      5 user ...
Models of user engagement (I)     Models based on engagement metrics•    6 general models•    Popularity, activity and loy...
Models of user engagement (II)   Models based on engagement metrics, user and time User-based [7 models]                  ...
Relationships between models Groups of models are independent from each other                       General       User   ...
Recap & Next  User engagement is complex and standard   metrics capture only a part of it  User engagement depends on user...
Understanding the problem:        Users on Yahoo! network of sites                                           - 25 -
Networked user engagement:       engagement across a network of sites   Large online providers (AOL, Google, Yahoo!, etc.)...
Online multi-tasking                                                       leaving a site is                              ...
Online multi-tasking          Users switch between sites within an online           session (several sites are visited and...
Online multi-tasking          Users switch between sites within an online           session (several sites are visited and...
Online multi-tasking  Users switch between sites within an online session   (several sites are visited and the same site i...
Networked user engagement - Two studies o  Is there a network effect?    o  study of 50 Yahoo! sites    o  downstream enga...
Is there a network effect? The success of a web site largely depends on itself, but              also on the network effec...
Measuring networked user engagement:                Downstream engagement     Downstream engagement            for site A ...
Downstream engagement                                      70%Varies significantly across sites     60%                   ...
What causes engagement to change?Web page style?                        Web page                         content?         ...
Methodology Front pagesof 50 popularY! properties     Definingcrawled every   downstream    hour        engagement        ...
DataPage attributes were defined for 50 popular (by page views) Yahoo! sites:    Sampled the front page once every hour du...
Page stylistics provide good information to predictdownstream engagement for many Yahoo! sitesThe top-10 sites for which d...
Influential features         o Time of day         o Number of (non-image/non-video) links to Yahoo! sites in HTML body   ...
Three case studies  Here we look at three different Yahoo! sites, and the effect of  their stylistics for downstream engag...
Influencing engagement through links   The correlations between the number of various links and the values of   downstream...
Users are more amenable to enhancingdownstream engagement during certain sessions Goal-specific sessions are those session...
Further Work: Quantifying the Network Effect Previously using one metric (downstream engagement), we   showed that there i...
Thank youQuestions?             Thanks to many people at Yahoo! Labs                                                - 44 -
Upcoming SlideShare
Loading in …5
×

User Engagement: from Sites to a Network of Sites or The Network Effect Matters!

2,433 views

Published on

Published in: Technology, Design
  • Be the first to comment

User Engagement: from Sites to a Network of Sites or The Network Effect Matters!

  1. 1. User Engagement:from Sites to a Network of Sites or The Network Effect Matters! Ricardo Baeza-Yates Mounia Lalmas Yahoo! Labs Barcelona Joint work with Janette Lehmann and Elad Yom-Tov and many others at Yahoo! Labs -1-
  2. 2. Outlineo  Motivation, definition and scopeo  Models of user engagemento  Networked user engagement -2-
  3. 3. Motivation, Definition and Scopeo  Definition and scopeo  Characteristics of user engagemento  Measures of user engagemento  Our research agenda -3-
  4. 4. User Engagement – connecting three sides Quality of the user experience that emphasizes the positive aspects of the interaction, and in particular the phenomena associated with users wanting to use a web application longer and frequently. Successful technologies are not just used, they are engaged withuser feelings: happy, sad, user mental states: concentrated, user interactions: click, readexcited, bored, … challenged, lost, interested … comment, recommend, buy, … The emotional, cognitive and/or behavioural connection that exists, at any point in time and over time, between a user and a technological resource S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011. -4-
  5. 5. Would a user engage with this web site? http://www.nhm.ac.uk/ -5-
  6. 6. Would a user engage with this web site? http://www.amazingthings.org/ (art event calendar) -6-
  7. 7. Would a user engage with this web site? http://www.lowpriceskates.com/ (e-commerce – skating) -7-
  8. 8. Would a user engage with this web site? http://chiptune.com/ (music repository) -8-
  9. 9. Would a user engage with this web site? http://www.theosbrinkagency.com/ (photographer) -9-
  10. 10. Characteristics of user engagement (I) •  Users must be focused to be engaged Focused attention •  Distortions in the subjective perception of time used to measure it •  Emotions experienced by user are intrinsically motivating Positive Affect •  Initial affective hook can induce a desire for exploration, active discovery or participation •  Sensory, visual appeal of interface stimulates, promote Aesthetics focused attention •  Linked to design principles (e.g. symmetry, balance, saliency) •  People remember enjoyable, useful, engaging experiences and want to repeat them Endurability •  Reflected in e.g. the propensity of users to recommend an experience/a site/a product - 10 -
  11. 11. Characteristics of user engagement (II) •  Novelty, surprise, unfamiliarity and unexpected Novelty •  Appeal to user curiosity, encourages inquisitive behavior and promotes repeated engagement •  Richness captures the growth potential of an activity Richness and control •  Control captures the extent to which a person is able to achieve this growth potential •  Trust is a necessary condition for user engagement Reputation, trust and •  Implicit contract among people and entities which is expectation more than technological Motivation, interests, incentives, and •  Difficulties in setting up “laboratory” style experiments benefits - 11 -
  12. 12. Forrester Research – The four I’s •  Presence of a user Involvement •  Measured by e.g. number of visitors, time spent •  Action of a user Interaction •  Measured by e.g. CTR, online transaction, uploaded photos or videos •  Affection or aversion of a user Intimacy •  Measured by e.g. satisfaction rating, sentiment analysis in blogs, comments, surveys, questionnaires •  Likelihood a user advocates Influence •  Measured by e.g. forwarded content, invitation to join Measuring Engagement, Forrester Research, June 2008- - 12
  13. 13. Peterson et al Engagement measure - 8 indices Click Depth Index: page views Duration Index: time spent Recency Index: rate at which users return over time Loyalty Index: level of long-term interaction the user has with the site or product (frequency) Brand Index: apparent awareness of the user of the brand, site, or product (search terms) Feedback Index: qualitative information including propensity to solicit additional information or supply direct feedback Interaction Index: user interaction with site or product (click, upload, transaction)Peterson etal. Measuring the immeasurable: visitor engagement, WebAnalyticsDemystified, September 2008 - 13 -
  14. 14. Measuring user engagement Measures   Characteristics   Self-reported Questionnaire, interview, report, Subjective, engagement   product reaction cards   user study (lab/online) Mostly qualitative Cognitive Task-based methods (time spent, Objective, engagement   follow-on task) user study (lab/online) Neurological measures (e.g. EEG) Mostly quantitative Physiological measures (e.g. eye Scalability an issue? tracking, mouse-tracking)   Interaction Web analytics + “data science” Objective, engagement   (CTR, bounce rate, dwell time, etc) data study Metrics and user models   Quantitative Large scale - 14 -
  15. 15. Interaction engagement – Online metricsProxy of user engagement - 15 -
  16. 16. Diagnostic and what we can doDiagnostic: work exists, but fragmented.In particular: o  What and how to measure depend on services and goals o  Going beyond site engagement What we have done: 1.  Models of user engagement 2.  Networked user engagement 3.  Complex networks analysis Future: Economic model for networked UE - 16 -
  17. 17. Models of User Engagement Online sites differ concerning their engagement! Games Search Users spend Users come much time per frequently and visit do not stay long Social media Special Users come Users come on frequently and average once stay long Service News Users visit site, Users come when needed periodically is it possible to model these differences and compare different classes of sites? - 17 -
  18. 18. Data and MetricsInteraction data, 2M users, July 2011, 80 USA sites Popularity #Users Number of distinct users #Visits Number of visits #Clicks Number of clicks Activity ClickDepth Average number of page views per visit. DwellTimeA Average time per visit Loyalty ActiveDays Number of days a user visited the site ReturnRate Number of times a user visited the site DwellTimeL Average time a user spend on the site. - 18 -
  19. 19. Diversity in User Engagement Engagement of a site depends on users and timeUsers and Loyalty Time and PopularitySites have different user groups Site engagement can be periodic or contains peaksProportion of user groups is site- dependent mail, social media media (special events) media, entertainment daily activity, shopping, navigation entertainment - 19 -
  20. 20. Methodology General models User-based models Time-based modelsDimensions 5 user groups weekdays, weekend 8 metrics 8 metrics per user 8 metrics per time span group#Dimensions 8 40 16 Kernel k-means with Kendall tau rank correlation kernel Num. of clusters based on eigenvalue distribution of kernel matrix Significant metric values with Kruskal-Wallis/Bonferonni#Clusters(Models) 6 7 5 Analyzing cluster centroids = models - 20 -
  21. 21. Models of user engagement (I) Models based on engagement metrics•  6 general models•  Popularity, activity and loyalty are independent from each other•  Popularity and loyalty are influenced by external and internal factors   e.g. frequency of publishing new information, events, personal interests•  Activity depends on the structure of the site interest- specific periodic media e-commerce, configuration - 21 -
  22. 22. Models of user engagement (II) Models based on engagement metrics, user and time User-based [7 models] Time-based [5 models] Models based on engagement per user Models based on engagement over group weekdays and weekendnavigation game, sport hobbies, daily news interest-specific   Sites of the same type (e.g. mainstream media) do not necessarily belong to the same model   The groups of models describe different aspects of engagement, i.e. they are independent from each other - 22 -
  23. 23. Relationships between models Groups of models are independent from each other General User Time General 0.00 3.50 4.23 User 3.50 0.00 4.25 Variance of Information Time 4.23 4.25 0.00 [0,5.61] Example:  Model mu2 [high popularity and activity in all user groups, increasing loyalty]  50% to model mt2 [high popularity on weekends and high loyalty on weekdays]  50% to model mt3 [high activity and loyalty on weekends] - 23 -
  24. 24. Recap & Next User engagement is complex and standard metrics capture only a part of it User engagement depends on users and time First step towards a taxonomy of models of user engagement … and associated metrics Next Interaction between models User demographics, time of the day, geo-location, etcJ. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012. - 24 -
  25. 25. Understanding the problem: Users on Yahoo! network of sites - 25 -
  26. 26. Networked user engagement: engagement across a network of sites Large online providers (AOL, Google, Yahoo!, etc.) offer not one service (site), but a network of services (sites) Each service is usually optimized individually, with some effort to direct users between them  Success of a service depends on itself, but also on how it is reached from other services (user traffic) Measuring user engagement across a network of sites should account for user traffic between sites. - 26 -
  27. 27. Online multi-tasking leaving a site is not a “bad thing!”users spend more and more of their online session multi-tasking, e.g. emailing,reading news, searching for information  ONLINE MULTI-TASKING navigating between sites, using browser tabs, bookmarks, etc seamless integration of social networks platforms into many services - 27 -
  28. 28. Online multi-tasking Users switch between sites within an online session (several sites are visited and the same site is visited several times)Navigation Back Link Other1 Browser usage changed button Less usage of back button1995 35.7% 45.7% 18.6%1997 31.7% 43.4% 24.9%2006 14.3% 43.5% 42.2% Oberdorf et al 1)  Usage of tabs, bookmarks, typing the URL directly,… 2)  http://uxmovement.com/navigation/why-external-links-should-open-in-new-tabs/ - 28 -
  29. 29. Online multi-tasking Users switch between sites within an online session (several sites are visited and the same site is visited several times)Navigation Browser usage changed More and more usage of tabs Back Link Other1 button1995 35.7% 45.7% 18.6%1997 31.7% 43.4% 24.9%2006 14.3% 43.5% 42.2% Dubroy et al Oberdorf et al] UX Movement2: External links affect your site and users Links that take users to different websites should open in new tabs. 1)  Usage of tabs, bookmarks, typing the URL directly,… 2)  http://uxmovement.com/navigation/why-external-links-should-open-in-new-tabs/ - 29 -
  30. 30. Online multi-tasking Users switch between sites within an online session (several sites are visited and the same site is visited several times) Navigation Back Link Other1 button 1995 35.7% 45.7% 18.6% 1997 31.7% 43.4% 24.9% 2006 14.3% 43.5% 42.2%   A short visit does not mean less engagement Measuring user engagement across a network of sites should account for multi-tasking - 30 -
  31. 31. Networked user engagement - Two studies o  Is there a network effect? o  study of 50 Yahoo! sites o  downstream engagement as a measure of networked user engagement o  effect of stylistics (layout and structure) o  Can we quantify the network effect? o  study of 728 Yahoo! sites and traffic between them o  use metrics from the complex network area together with engagement metrics to characterize networked user engagement - 31 -
  32. 32. Is there a network effect? The success of a web site largely depends on itself, but also on the network effect This is particularly relevant for the Yahoo! network of propertieso  Can we measure the network effect? o  Downstream engagemento  Can we influence downstream engagement? o  Session types o  Link types - 32 -
  33. 33. Measuring networked user engagement: Downstream engagement Downstream engagement for site A (% remaining session time) Site A tes o! si o Yah User session - 33 -
  34. 34. Downstream engagement 70%Varies significantly across sites 60% 50% 40% 30%Exhibits different distributions 20% 10% according to site type 0% 0.12 0.1 0.08Is not highly correlated with other 0.06 0.04 engagement measures such as 0.02 dwell time 0 1% 9% 17% 25% 33% 41% 49% 58% 66% 74% 82% 90% 98% 140 120 100Optimizing downstream engagement 80 60 will have little effect on user 40 20 engagement within that site 0 0% 20% 40% 60% 80% - 34 -
  35. 35. What causes engagement to change?Web page style? Web page content? - 35 -
  36. 36. Methodology Front pagesof 50 popularY! properties Definingcrawled every downstream hour engagement measure Influencing Studying theSample user network effect of links data from Measuring downstream on fronttoolbar data downstream engagement pages on engagement with front (19.4M downstream sessions, measure with page engagement265K users) web page stylistics stylistics and content May 2011 - 36 -
  37. 37. DataPage attributes were defined for 50 popular (by page views) Yahoo! sites: Sampled the front page once every hour during the month of May 2011. Generate two types of attributes for each site at each time and date: Stylistics (layout and structure) of a page General such as time of day, date, weekday or notDownstream engagement values were measured using Yahoo! toolbar data: A total of 19.4M sessions recorded from approximately 265,000 users.User and front page datasets joined by site, date and time, such that for each site and each date and time combination we have: average downstream engagement average dwell time vector of corresponding style attributes collected around the time that user engagement was measured - 37 -
  38. 38. Page stylistics provide good information to predictdownstream engagement for many Yahoo! sitesThe top-10 sites for which downstream engagement (DE) could be “accurately” predictedbased on their stylistics Accuracy Precision Average DE site 1 0.80 0.54 0.07 site 2 0.76 0.52 0.11 site 3 0.72 0.43 0.21 site 4 0.71 0.40 0.18 site 5 0.65 0.42 0.20 site 6 0.63 0.34 0.19 site 7 0.63 0.38 0.26 site 8 0.63 0.44 0.14 site 9 0.61 0.34 0.18 site 10 0.60 0.31 0.13 Downstream engagement of a number of sites of not particular types (models) could not be predicted from their stylistics. - 38 -
  39. 39. Influential features o Time of day o Number of (non-image/non-video) links to Yahoo! sites in HTML body o Average rank of Yahoo! links on page o Number of (non-image/non-video) links to non-Yahoo! sites in HTML body o Number of span tags (tags that allow adding style to content or manipulating content, e.g. JavaScript)o  Link  placements  and  number  of  Yahoo!  links  can  influence  downstream  engagement   o  Not  new,  but  here  shown  to  hold  also  across  sites    o  Links  to  non-­‐Yahoo!  sites  have  a  posi>ve  effect  on  downstream  engagement   o  Possibly  because  when  users  are  faced  with  abundance  of  outside  links  they  decide  to   focus  their  aBen>on  on  a  central  content  provider,  rather  than  visi>ng  mul>tude  of   external  sites     - 39 -
  40. 40. Three case studies Here we look at three different Yahoo! sites, and the effect of their stylistics for downstream engagement Sites Average Accuracy Precision downstream engagement e-commerce 0.26 (+/- 0.31) 0.63 0.38 news 0.15 (+/- 0.02) 0.65 0.37 women-interests 0.21 (+/- 0.06) 0.72 0.53 Number of unique Yahoo! links (-)Time of day (+) Number of image links to non-Yahoo! Number of (non-image/non-video) linksWeekend (-) sites in the body of the HTML (-) to Yahoo sites in the body of theNumber of unique Yahoo! links (-) Number of table elements (-) HTML (+)Average rank of Yahoo! links on page (-) Average rank of Yahoo! links on page (-) Number of (non-image/non-video) linksNumber of paragraph tags (+) Number of (non-image/non-video) links to non-Yahoo! sites in the body of to non-Yahoo! sites in the body of the HTML (+) the HTML (+) Number of video links within the page (+) Time of day (+) Number of Java scripts on the page (-) - 40 -
  41. 41. Influencing engagement through links The correlations between the number of various links and the values of downstream engagement and dwell time• e-commerce: • links have little effect on e-commerce news women- downstream engagement, but interests have on dwell time Downstream engagement• news: • more news stories lead to more Same site 0.03 -0.31 -0.27 time spent on news Other Y! site -0.09 0.20 0.22 • external links do not affect Non Y! site -0.10 0.04 -0.25 downstream engagement, but Dwell time affect dwell time Same site 0.51 0.78 0.82• women-interests: Other Y! site -0.61 0.38 -0.68 • links to other Yahoo! sites can Non Y! site -0.51 0.04 0.80 help increase engagement, but they may decrease dwell time - 41 -
  42. 42. Users are more amenable to enhancingdownstream engagement during certain sessions Goal-specific sessions are those sessions where users have a specific goal in mind: do email, read news, check FB Sessions when at least 50% of visited sites belonged to the five most common sites (for that user) were classified as goal- specific Goal-specific sessions accounted for 38% of sessions. Approximately 92% of users had sessions of both kinds. Average downstream engagement in goal-specific sessions was 0.16 vs. 0.2 for other sessions. Accuracy of predicting downstream engagement was 0.76 for goal-specific sessions vs. 0.81 for other sessions. When users do not have specific goals in mind, they may be more ready to accept suggestions (e.g. more links) for additional browsing - 42 -
  43. 43. Further Work: Quantifying the Network Effect Previously using one metric (downstream engagement), we showed that there is a network effect, and that the network effect can be influenced. We go one step further and propose a methodology to account for the traffic between sites when measuring user engagement on a network of sites. o  Engagement networks o  Metrics from complex networks area (network-level and node- level) o  Application on 728 Yahoo! sites - 43 -
  44. 44. Thank youQuestions? Thanks to many people at Yahoo! Labs - 44 -

×