SlideShare a Scribd company logo
1 of 40
Download to read offline
Measuring Web User Engagement:
   a cauldron of web analytics, focus attention,
         positive affect, user interest, saliency,
             mouse movement & multi-tasking



                                 Mounia Lalmas
                                  Yahoo! Labs
                                     Barcelona
Click-through rate as proxy of user
engagement!
Click-through rate as proxy of
        relevance!




                                                                                       Multimedia search
                                                                                       activities often
                                                                                       driven by
                                                                                       entertainment
                                                                                       needs, not by
M. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011.
                                                                                       information needs
Click-through rate as proxy of user
user satisfaction!




     I just wanted the phone number … I am totally satisfied 
In this talk – results, messages
  & questions

1. Big data and in-depth focused user studies “a
   must”!
2. Users “multi-task” online, what does this
   mean?
3. Mouse movement hard to “experiment with”
   and/or “interpret”.
4. Using crowd-sourcing “I think” worked fine.
This talk is not about aesthetics
                                                        … but see later




                                http://www.lowpriceskates.com/ (e-commerce – skating)

Source: http://www.webpagesthatsuck.com/
This talk is not about usability




                                           http://chiptune.com/ (music repository)

Source: http://www.webpagesthatsuck.com/
User Engagement – connecting three sides
       User engagement is a quality of the user experience
        that emphasizes the positive aspects of interaction – in
        particular the fact of being captivated by the technology.

       Successful technologies are not just used, they are
        engaged with.

user feelings: happy, sad,                     user mental states: concentrated,                        user interactions: click, read
excited, …                                     lost, involved, …                                        comment, recommend, buy, …



         The emotional, cognitive and behavioural connection
         that exists, at any point in time and over time, between a
         user and a technological resource

S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper),
WSDM Workshop on User Modelling for Web Applications, 2011.
Characteristics of user engagement

     Focused Attention                                       Novelty


                  Positive Affect                                Richness & Control

            Aesthetics
                                                         Reputation, Trust &
                                                            Expectation
                        Endurability
                                                                              Motivation,
                                                                         Interests, Incentives
                                                                              & Benefits


H.L. O'Brien. Defining and Measuring Engagement in User Experiences with Technology. PhD Thesis,
2008.
H.L. O'Brien & E.G. Toms. JASIST 2008, JASIST 2010.
Measuring user engagement
Connecting three measurement approaches
                                                                 nt
                                                            ageme
                                                 ti   on eng
                                          interac

USER ENGAGEMENT




    self-
         repo
             r te   d en
                           gage                                      ment
                               m en
                                      t                      n   gage
                                                         e
                                              n   it ive
                                           cog
Models of user engagement …
towards a taxonomy?
                                                                 nt
                                                            ageme
                                                 ti   on eng
                                          interac

USER ENGAGEMENT




    self-
         repo
             r te   d en
                           gage                                      ment
                               m en
                                      t                      n   gage
                                                         e
                                              n   it ive
                                           cog
Models of user engagement
   Online sites differ concerning their engagement!


             Games                          Search
             Users spend                    Users come
             much time per                  frequently and do
             visit                          not stay long


             Social media                   Special
             Users come                     Users come on
             frequently and                 average once
             stay long


             Service                        News
             Users visit site,              Users come
             when needed                    periodically
Data and Metrics
Interaction data, 2M users, July 2011, 80 US sites
 Popularity   #Users       Number of distinct users


              #Visits      Number of visits


              #Clicks      Number of clicks


 Activity     ClickDepth   Average number of page views per visit.


              DwellTimeA   Average time per visit


 Loyalty      ActiveDays   Number of days a user visited the site


              ReturnRate   Number of times a user visited the site


              DwellTimeL   Average time a user spend on the site.
Methodology
                 General models                    Time-based models
Dimensions                                         weekdays, weekend
                     8 metrics                   8 metrics per time span
#Dimensions              8                                  16

                                   Kernel k-means with
                             Kendall tau rank correlation kernel
              Nb of clusters based on eigenvalue distribution of kernel matrix
                 Significant metric values with Kruskal-Wallis/Bonferonni

#Clusters
(Models)                 6                                  5


                             Analysing cluster centroids = models
Models of user engagement
[6 general]
•   Popularity, activity and loyalty are independent from each other
•   Popularity and loyalty are influenced by external and internal factors
      e.g. frequency of publishing new information, events, personal
        interests
•   Activity depends on the structure of the site
      interest-specific

periodic
media

e-commerce

media (daily)
search


                    models based on engagement metrics only
Time-based [5 models]
  Models based on engagement over weekdays and weekend


                      work-related
daily news




  hobbies,
  interest-specific
  weather
              time-based models ≠ general models

             next put all and more together! let machine learning tell you more!
Models of user engagement –
  Recap & Next
 User engagement is complex and standard
  metrics capture only a part of it
 User engagement depends on time (and users)
 First step towards a taxonomy of models of user
  engagement … and associated metrics

 Next
        More sites, more models
        User demographics, time of the day, geo-location,
         etc.
        Online multi-tasking
J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.
Online multi-tasking
                                                                 nt
                                                            ageme
                                                 ti   on eng
                                          interac

USER ENGAGEMENT




    self-
         repo
             r te   d en
                           gage                                      ment
                               m en
                                      t                      n   gage
                                                         e
                                              n   it ive
                                           cog
Online multi-tasking
                                                                 181K users, 2 months browser
                                                                 data, 600 sites, 4.8M sessions

                                                                 •only 40% of the sessions have
                                                                 no site revisitation

                                                                 •hyperlinking, backpaging and
                                                                 teleporting


                                                                  leaving a site is
                                                                  not a “bad thing!”

(fictitious navigation between sites within an online session)


  users spend more and more of their online session multi-tasking, e.g. emailing,
  reading news, searching for information  ONLINE MULTI-TASKING
           navigating between sites, using browser tabs, bookmarks, etc
           seamless integration of social networks platforms into many services
Navigating between sites –
hyperliking, backpaging and teleporting




Number of backpaging actions is an under-estimate!
Revisitation and navigation patterns
Online multi-tasking – Some results
48% sites visited at least 9 times
Revisitation “level” depends on site

10% users accessed a site 9+ times (23% for search
 sites); 28% at least four times (44% for search sites)
Activity on site decreases with each revisit but
 activity on many search and adult sites increases

Backpaging usually increases with each revisit but
 hyperlinking remains important means to navigate
 between sites
Online multi-tasking –
      Recap & Next




J. Lehmann, M. Lalmas & G. Dupret. Online Multi-Tasking and User Engagement. Submitted for publication, 2013.
Focus attention, positive affect & saliency
                                                                  nt
                                                             ageme
                                                  ti   on eng
                                           interac

USER ENGAGEMENT




     self-
          repo
              r te   d en
                            gage                                      ment
                                m en
                                       t                      n   gage
                                                          e
                                               n   it ive
                                            cog
Saliency, attention and positive affect

How the visual catchiness (saliency) of
 “relevant” information impacts user
 engagement metrics such as focused
 attention and emotion (affect)
  focused attention refers to the exclusion of
   other things
  affect relates to the emotions experienced
   during the interaction
Saliency model of visual attention
 developed by Itti and Koch
 L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual
 attention. Vision Research, 40, 2000.
Manipulating saliency
 non-salient condition
 salient condition




                         Web page screenshot   Saliency maps
Study design
 8 tasks = finding latest news or headline on celebrity or
  entertainment topic
 Affect measured pre- and post- task using the Positive
  e.g. “determined”, “attentive” and Negative e.g. “hostile”, “afraid”
  Affect Schedule (PANAS)
 Focused attention measured with 7-item focused
  attention subscale e.g. “I was so involved in my news tasks that I
   lost track of time”, “I blocked things out around me when I was
   completing the news tasks” and perceived time
 Interest level in topics (pre-task) and questionnaire
  (post-task) e.g. “I was interested in the content of the web pages”,
   “I wanted to find out more about the topics that I encountered on the
   web pages”
 189 (90+99) participants from Amazon Mechanical Turk
Saliency and positive affect
When headlines are visually non-salient
   users are slow at finding them, report more
   distraction due to web page features, and show a
   drop in affect
When headlines are visually catchy or
 salient
   user find them faster, report that it is easy to
   focus, and maintain positive affect


Saliency is helpful in task performance,
 focusing/avoiding distraction and in
Saliency and focused attention
Adapted focused attention subscale from the online
 shopping domain to entertainment news domain

Users reported “easier to focus in the salient
 condition” BUT no significant improvement in the
 focused attention subscale or differences in
 perceived time spent on tasks

User interest in web page content is a good
 predictor of focused attention, which in turn is a
 good predictor of positive affect
Saliency and user engagement –
    Recap & Next
  Interaction of saliency, focused attention,
   and affect, together with user interest, is
   complex
  Next:
        include web page content as a quality of user
         engagement in focused attention scale
        more “realistic” user (interactive) reading
         experience
        bio-metrics (mouse-tracking, eye-tracking, facial
         expression, etc)
L. McCay-Peet, M. Lalmas, V. Navalpakkam. On saliency, affect and focused attention, CHI 2012
Mouse tracking, positive effect, attention
                                                                  nt
                                                             ageme
                                                  ti   on eng
                                           interac

USER ENGAGEMENT




     self-
          repo
              r te   d en
                            gage                                      ment
                                m en
                                       t                      n   gage
                                                          e
                                               n   it ive
                                            cog
Mouse tracking … and user engagement
  324 users from Amazon Mechanical Turk
   (between subject design)
  Two domains (BBC and Wikipedia)
  Two tasks (reading and quiz)
  “Normal vs Ugly” interface
  Questionnaires (qualitative data)
    focus attention, positive effect, novelty, interest,
     usability, aesthetics
    + demographics, handeness & hardware
  Mouse tracking (quantitative data)
    movement speed, movement rate, click rate, pause
     length, percentage of time still
“Ugly” vs “Normal” Interface (BBC News)
“Ugly” vs “Normal” (Wikipedia)
Mouse tracking can tell about
Age

Hardware
 Mouse
 Trackpad


Task
  Searching: There are many different types of phobia. What is
   Gephyrophobia a fear of?
  Reading: (Wikipedia) Archimedes, Section 1: Biography
Mouse tracking could not tell much on
Mouse tracking and user engagement —
 Recap & Next
 High level of ecological validity
 Age, task, and hardware
 Do we have a Hawthorne Effect???
 “Usability” vs engagement
      “Even uglier” interface? I don’t think so
 Within- vs between-subject design?
 Next
      Sequence of movements
      Automatic clustering
D. Warnock and M. Lalmas. An Exploration of Cursor tracking Data. Submitted for publication, 2013.
Connecting three measurement approaches
                                     on
                           interacti

   The value of a click?




      self-
           repo
               r te   d

                                        e
                                 gn itiv
                            co
Thank you



                          Collaborators:
                             Ioannis Arapakis
                             Ricardo Baeza-Yates
                             Georges Dupret
                             Janette Lehmann
                             Lori McCay-Peet (Dalhousie University)
                             Vidhya Navalpakkam
                             David Warnock (Glasgow University)
                             Elad Yom-Tov
Contact: mounia@acm.org
                             and many others at Yahoo! Labs

More Related Content

More from Mounia Lalmas-Roelleke

Personalizing the listening experience
Personalizing the listening experiencePersonalizing the listening experience
Personalizing the listening experienceMounia Lalmas-Roelleke
 
Recommending and Searching (Research @ Spotify)
Recommending and Searching (Research @ Spotify)Recommending and Searching (Research @ Spotify)
Recommending and Searching (Research @ Spotify)Mounia Lalmas-Roelleke
 
Tutorial on metrics of user engagement -- Applications to Search & E- commerce
Tutorial on metrics of user engagement -- Applications to Search & E- commerceTutorial on metrics of user engagement -- Applications to Search & E- commerce
Tutorial on metrics of user engagement -- Applications to Search & E- commerceMounia Lalmas-Roelleke
 
An introduction to system-oriented evaluation in Information Retrieval
An introduction to system-oriented evaluation in Information RetrievalAn introduction to system-oriented evaluation in Information Retrieval
An introduction to system-oriented evaluation in Information RetrievalMounia Lalmas-Roelleke
 
Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...
Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...
Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...Mounia Lalmas-Roelleke
 
Social Media and AI: Don’t forget the users
Social Media and AI: Don’t forget the usersSocial Media and AI: Don’t forget the users
Social Media and AI: Don’t forget the usersMounia Lalmas-Roelleke
 
Describing Patterns and Disruptions in Large Scale Mobile App Usage Data
Describing Patterns and Disruptions in Large Scale Mobile App Usage DataDescribing Patterns and Disruptions in Large Scale Mobile App Usage Data
Describing Patterns and Disruptions in Large Scale Mobile App Usage DataMounia Lalmas-Roelleke
 
Story-focused Reading in Online News and its Potential for User Engagement
Story-focused Reading in Online News and its Potential for User EngagementStory-focused Reading in Online News and its Potential for User Engagement
Story-focused Reading in Online News and its Potential for User EngagementMounia Lalmas-Roelleke
 
Mobile advertising: The preclick experience
Mobile advertising: The preclick experienceMobile advertising: The preclick experience
Mobile advertising: The preclick experienceMounia Lalmas-Roelleke
 
Predicting Pre-click Quality for Native Advertisements
Predicting Pre-click Quality for Native AdvertisementsPredicting Pre-click Quality for Native Advertisements
Predicting Pre-click Quality for Native AdvertisementsMounia Lalmas-Roelleke
 
Improving Post-Click User Engagement on Native Ads via Survival Analysis
Improving Post-Click User Engagement on Native Ads via Survival AnalysisImproving Post-Click User Engagement on Native Ads via Survival Analysis
Improving Post-Click User Engagement on Native Ads via Survival AnalysisMounia Lalmas-Roelleke
 
A Journey into Evaluation: from Retrieval Effectiveness to User Engagement
A Journey into Evaluation: from Retrieval Effectiveness to User EngagementA Journey into Evaluation: from Retrieval Effectiveness to User Engagement
A Journey into Evaluation: from Retrieval Effectiveness to User EngagementMounia Lalmas-Roelleke
 
Promoting Positive Post-click Experience for In-Stream Yahoo Gemini Users
Promoting Positive Post-click Experience for In-Stream Yahoo Gemini UsersPromoting Positive Post-click Experience for In-Stream Yahoo Gemini Users
Promoting Positive Post-click Experience for In-Stream Yahoo Gemini UsersMounia Lalmas-Roelleke
 
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity SearchFrom “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity SearchMounia Lalmas-Roelleke
 
How Big Data is Changing User Engagement
How Big Data is Changing User EngagementHow Big Data is Changing User Engagement
How Big Data is Changing User EngagementMounia Lalmas-Roelleke
 
Measuring user engagement: the do, the do not do, and the we do not know
Measuring user engagement: the do, the do not do, and the we do not knowMeasuring user engagement: the do, the do not do, and the we do not know
Measuring user engagement: the do, the do not do, and the we do not knowMounia Lalmas-Roelleke
 
An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...Mounia Lalmas-Roelleke
 

More from Mounia Lalmas-Roelleke (20)

Recommending and searching @ Spotify
Recommending and searching @ SpotifyRecommending and searching @ Spotify
Recommending and searching @ Spotify
 
Personalizing the listening experience
Personalizing the listening experiencePersonalizing the listening experience
Personalizing the listening experience
 
Recommending and Searching (Research @ Spotify)
Recommending and Searching (Research @ Spotify)Recommending and Searching (Research @ Spotify)
Recommending and Searching (Research @ Spotify)
 
Search @ Spotify
Search @ Spotify Search @ Spotify
Search @ Spotify
 
Tutorial on metrics of user engagement -- Applications to Search & E- commerce
Tutorial on metrics of user engagement -- Applications to Search & E- commerceTutorial on metrics of user engagement -- Applications to Search & E- commerce
Tutorial on metrics of user engagement -- Applications to Search & E- commerce
 
An introduction to system-oriented evaluation in Information Retrieval
An introduction to system-oriented evaluation in Information RetrievalAn introduction to system-oriented evaluation in Information Retrieval
An introduction to system-oriented evaluation in Information Retrieval
 
Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...
Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...
Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...
 
Social Media and AI: Don’t forget the users
Social Media and AI: Don’t forget the usersSocial Media and AI: Don’t forget the users
Social Media and AI: Don’t forget the users
 
Advertising Quality Science
Advertising Quality ScienceAdvertising Quality Science
Advertising Quality Science
 
Describing Patterns and Disruptions in Large Scale Mobile App Usage Data
Describing Patterns and Disruptions in Large Scale Mobile App Usage DataDescribing Patterns and Disruptions in Large Scale Mobile App Usage Data
Describing Patterns and Disruptions in Large Scale Mobile App Usage Data
 
Story-focused Reading in Online News and its Potential for User Engagement
Story-focused Reading in Online News and its Potential for User EngagementStory-focused Reading in Online News and its Potential for User Engagement
Story-focused Reading in Online News and its Potential for User Engagement
 
Mobile advertising: The preclick experience
Mobile advertising: The preclick experienceMobile advertising: The preclick experience
Mobile advertising: The preclick experience
 
Predicting Pre-click Quality for Native Advertisements
Predicting Pre-click Quality for Native AdvertisementsPredicting Pre-click Quality for Native Advertisements
Predicting Pre-click Quality for Native Advertisements
 
Improving Post-Click User Engagement on Native Ads via Survival Analysis
Improving Post-Click User Engagement on Native Ads via Survival AnalysisImproving Post-Click User Engagement on Native Ads via Survival Analysis
Improving Post-Click User Engagement on Native Ads via Survival Analysis
 
A Journey into Evaluation: from Retrieval Effectiveness to User Engagement
A Journey into Evaluation: from Retrieval Effectiveness to User EngagementA Journey into Evaluation: from Retrieval Effectiveness to User Engagement
A Journey into Evaluation: from Retrieval Effectiveness to User Engagement
 
Promoting Positive Post-click Experience for In-Stream Yahoo Gemini Users
Promoting Positive Post-click Experience for In-Stream Yahoo Gemini UsersPromoting Positive Post-click Experience for In-Stream Yahoo Gemini Users
Promoting Positive Post-click Experience for In-Stream Yahoo Gemini Users
 
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity SearchFrom “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
From “Selena Gomez” to “Marlon Brando”: Understanding Explorative Entity Search
 
How Big Data is Changing User Engagement
How Big Data is Changing User EngagementHow Big Data is Changing User Engagement
How Big Data is Changing User Engagement
 
Measuring user engagement: the do, the do not do, and the we do not know
Measuring user engagement: the do, the do not do, and the we do not knowMeasuring user engagement: the do, the do not do, and the we do not know
Measuring user engagement: the do, the do not do, and the we do not know
 
An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...An Engaging Click ... or how can user engagement measurement inform web searc...
An Engaging Click ... or how can user engagement measurement inform web searc...
 

Recently uploaded

Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_planJamie (Taka) Wang
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 

Recently uploaded (20)

Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
20200723_insight_release_plan
20200723_insight_release_plan20200723_insight_release_plan
20200723_insight_release_plan
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 

Measuring Web User Engagement: a cauldron of many things.

  • 1. Measuring Web User Engagement: a cauldron of web analytics, focus attention, positive affect, user interest, saliency, mouse movement & multi-tasking Mounia Lalmas Yahoo! Labs Barcelona
  • 2. Click-through rate as proxy of user engagement!
  • 3. Click-through rate as proxy of relevance! Multimedia search activities often driven by entertainment needs, not by M. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011. information needs
  • 4. Click-through rate as proxy of user user satisfaction! I just wanted the phone number … I am totally satisfied 
  • 5. In this talk – results, messages & questions 1. Big data and in-depth focused user studies “a must”! 2. Users “multi-task” online, what does this mean? 3. Mouse movement hard to “experiment with” and/or “interpret”. 4. Using crowd-sourcing “I think” worked fine.
  • 6. This talk is not about aesthetics … but see later http://www.lowpriceskates.com/ (e-commerce – skating) Source: http://www.webpagesthatsuck.com/
  • 7. This talk is not about usability http://chiptune.com/ (music repository) Source: http://www.webpagesthatsuck.com/
  • 8. User Engagement – connecting three sides  User engagement is a quality of the user experience that emphasizes the positive aspects of interaction – in particular the fact of being captivated by the technology.  Successful technologies are not just used, they are engaged with. user feelings: happy, sad, user mental states: concentrated, user interactions: click, read excited, … lost, involved, … comment, recommend, buy, … The emotional, cognitive and behavioural connection that exists, at any point in time and over time, between a user and a technological resource S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.
  • 9. Characteristics of user engagement Focused Attention Novelty Positive Affect Richness & Control Aesthetics Reputation, Trust & Expectation Endurability Motivation, Interests, Incentives & Benefits H.L. O'Brien. Defining and Measuring Engagement in User Experiences with Technology. PhD Thesis, 2008. H.L. O'Brien & E.G. Toms. JASIST 2008, JASIST 2010.
  • 11. Connecting three measurement approaches nt ageme ti on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog
  • 12. Models of user engagement … towards a taxonomy? nt ageme ti on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog
  • 13. Models of user engagement Online sites differ concerning their engagement! Games Search Users spend Users come much time per frequently and do visit not stay long Social media Special Users come Users come on frequently and average once stay long Service News Users visit site, Users come when needed periodically
  • 14. Data and Metrics Interaction data, 2M users, July 2011, 80 US sites Popularity #Users Number of distinct users #Visits Number of visits #Clicks Number of clicks Activity ClickDepth Average number of page views per visit. DwellTimeA Average time per visit Loyalty ActiveDays Number of days a user visited the site ReturnRate Number of times a user visited the site DwellTimeL Average time a user spend on the site.
  • 15. Methodology General models Time-based models Dimensions weekdays, weekend 8 metrics 8 metrics per time span #Dimensions 8 16 Kernel k-means with Kendall tau rank correlation kernel Nb of clusters based on eigenvalue distribution of kernel matrix Significant metric values with Kruskal-Wallis/Bonferonni #Clusters (Models) 6 5 Analysing cluster centroids = models
  • 16. Models of user engagement [6 general] • Popularity, activity and loyalty are independent from each other • Popularity and loyalty are influenced by external and internal factors  e.g. frequency of publishing new information, events, personal interests • Activity depends on the structure of the site interest-specific periodic media e-commerce media (daily) search models based on engagement metrics only
  • 17. Time-based [5 models] Models based on engagement over weekdays and weekend work-related daily news hobbies, interest-specific weather time-based models ≠ general models next put all and more together! let machine learning tell you more!
  • 18. Models of user engagement – Recap & Next User engagement is complex and standard metrics capture only a part of it User engagement depends on time (and users) First step towards a taxonomy of models of user engagement … and associated metrics Next  More sites, more models  User demographics, time of the day, geo-location, etc.  Online multi-tasking J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.
  • 19. Online multi-tasking nt ageme ti on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog
  • 20. Online multi-tasking 181K users, 2 months browser data, 600 sites, 4.8M sessions •only 40% of the sessions have no site revisitation •hyperlinking, backpaging and teleporting leaving a site is not a “bad thing!” (fictitious navigation between sites within an online session) users spend more and more of their online session multi-tasking, e.g. emailing, reading news, searching for information  ONLINE MULTI-TASKING navigating between sites, using browser tabs, bookmarks, etc seamless integration of social networks platforms into many services
  • 21. Navigating between sites – hyperliking, backpaging and teleporting Number of backpaging actions is an under-estimate!
  • 23. Online multi-tasking – Some results 48% sites visited at least 9 times Revisitation “level” depends on site 10% users accessed a site 9+ times (23% for search sites); 28% at least four times (44% for search sites) Activity on site decreases with each revisit but activity on many search and adult sites increases Backpaging usually increases with each revisit but hyperlinking remains important means to navigate between sites
  • 24. Online multi-tasking – Recap & Next J. Lehmann, M. Lalmas & G. Dupret. Online Multi-Tasking and User Engagement. Submitted for publication, 2013.
  • 25. Focus attention, positive affect & saliency nt ageme ti on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog
  • 26. Saliency, attention and positive affect How the visual catchiness (saliency) of “relevant” information impacts user engagement metrics such as focused attention and emotion (affect) focused attention refers to the exclusion of other things affect relates to the emotions experienced during the interaction Saliency model of visual attention developed by Itti and Koch L. Itti and C. Koch. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 2000.
  • 27. Manipulating saliency non-salient condition salient condition Web page screenshot Saliency maps
  • 28. Study design  8 tasks = finding latest news or headline on celebrity or entertainment topic  Affect measured pre- and post- task using the Positive e.g. “determined”, “attentive” and Negative e.g. “hostile”, “afraid” Affect Schedule (PANAS)  Focused attention measured with 7-item focused attention subscale e.g. “I was so involved in my news tasks that I lost track of time”, “I blocked things out around me when I was completing the news tasks” and perceived time  Interest level in topics (pre-task) and questionnaire (post-task) e.g. “I was interested in the content of the web pages”, “I wanted to find out more about the topics that I encountered on the web pages”  189 (90+99) participants from Amazon Mechanical Turk
  • 29. Saliency and positive affect When headlines are visually non-salient  users are slow at finding them, report more distraction due to web page features, and show a drop in affect When headlines are visually catchy or salient  user find them faster, report that it is easy to focus, and maintain positive affect Saliency is helpful in task performance, focusing/avoiding distraction and in
  • 30. Saliency and focused attention Adapted focused attention subscale from the online shopping domain to entertainment news domain Users reported “easier to focus in the salient condition” BUT no significant improvement in the focused attention subscale or differences in perceived time spent on tasks User interest in web page content is a good predictor of focused attention, which in turn is a good predictor of positive affect
  • 31. Saliency and user engagement – Recap & Next Interaction of saliency, focused attention, and affect, together with user interest, is complex Next: include web page content as a quality of user engagement in focused attention scale more “realistic” user (interactive) reading experience bio-metrics (mouse-tracking, eye-tracking, facial expression, etc) L. McCay-Peet, M. Lalmas, V. Navalpakkam. On saliency, affect and focused attention, CHI 2012
  • 32. Mouse tracking, positive effect, attention nt ageme ti on eng interac USER ENGAGEMENT self- repo r te d en gage ment m en t n gage e n it ive cog
  • 33. Mouse tracking … and user engagement  324 users from Amazon Mechanical Turk (between subject design)  Two domains (BBC and Wikipedia)  Two tasks (reading and quiz)  “Normal vs Ugly” interface  Questionnaires (qualitative data)  focus attention, positive effect, novelty, interest, usability, aesthetics  + demographics, handeness & hardware  Mouse tracking (quantitative data)  movement speed, movement rate, click rate, pause length, percentage of time still
  • 34. “Ugly” vs “Normal” Interface (BBC News)
  • 36. Mouse tracking can tell about Age Hardware Mouse Trackpad Task  Searching: There are many different types of phobia. What is Gephyrophobia a fear of?  Reading: (Wikipedia) Archimedes, Section 1: Biography
  • 37. Mouse tracking could not tell much on
  • 38. Mouse tracking and user engagement — Recap & Next High level of ecological validity Age, task, and hardware Do we have a Hawthorne Effect??? “Usability” vs engagement “Even uglier” interface? I don’t think so Within- vs between-subject design? Next Sequence of movements Automatic clustering D. Warnock and M. Lalmas. An Exploration of Cursor tracking Data. Submitted for publication, 2013.
  • 39. Connecting three measurement approaches on interacti The value of a click? self- repo r te d e gn itiv co
  • 40. Thank you Collaborators: Ioannis Arapakis Ricardo Baeza-Yates Georges Dupret Janette Lehmann Lori McCay-Peet (Dalhousie University) Vidhya Navalpakkam David Warnock (Glasgow University) Elad Yom-Tov Contact: mounia@acm.org and many others at Yahoo! Labs