Your SlideShare is downloading. ×
  • Like
An Engaging Click ... or how can user engagement measurement inform web search evaluation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

An Engaging Click ... or how can user engagement measurement inform web search evaluation

  • 1,798 views
Published

A good search engine is one when users come very regularly, type their queries, get their results, and leave quickly. With user engagement metrics from web analytics, these translate to a low dwell …

A good search engine is one when users come very regularly, type their queries, get their results, and leave quickly. With user engagement metrics from web analytics, these translate to a low dwell time, often low CTR, but a very high return rate. But user engagement is not just about this. User engagement is a complex phenomenon that requires a number of approaches for its measurement: we can ask the user about their experience though questionnaires, we can observe where they look or move the mouse, and we can calculate various web analytic metrics. The aim of this talk is to discuss how current work on user engagement, not necessary specific to web search, can provide insights into putting search into more broader perspectives.

This presentation is part of Search Solutions 2013, 27 November 2013, at the BCS HQ. A first version of this talk was given at the SIGIR 2013 Industry Day by Ricardo Baeza-Yates.

Published in Technology , Design
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,798
On SlideShare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
18
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. An Engaging Click Mounia Lalmas Ya h o o L a b s London Search Solutions 2013
  • 2. This talk What is user engagement? What are the characteristics of user engagement? How to measure user engagement? What is user engagement in web search? What is an engaging click? 1.  inter-session 2.  online multi-tasking 3.  downstream engagement 4.  serendipity work on user engagement across web applications implications to web search
  • 3. This talk What is user engagement? What are the characteristics of user engagement? How to measure user engagement? What is user engagement in web search? What is an engaging click? 1.  inter-session 2.  online multi-tasking 3.  downstream engagement 4.  serendipity Work on user engagement across web applications Implications to search
  • 4. What is user engagement? User engagement is a quality of the user experience that emphasizes the positive aspects of interaction – in particular the fact of being captivated by the technology (Attfield et al, 2011). user feelings: happy, sad, excited, … user mental states: flow, presence, immersion, … user interactions: click, read, comment, buy… emotional, cognitive and behavioural connection that exists, at any point in time and over time, between a user and a technological resource
  • 5. Why is it important to engage users? §  In today’s wired world, users have enhanced expectations about their interactions with technology … resulting in increased competition amongst the purveyors and designers of interactive systems. §  In addition to utilitarian factors, such as usability, we must consider the hedonic and experiential factors of interacting with technology, such as fun, fulfillment, play, and user engagement. (O’Brien, Lalmas & Yom-Tov, 2013)
  • 6. Characteristics of user engagement Endurability Aesthetics (Read, MacFarlane, & Casey, (Jacques et al, 1995; O’Brien, 2008) 2002; O’Brien, 2008) Motivation, interests, incentives, and benefits (Jacques et al., 1995; O’Brien & Toms, 2008) Focused attention (Webster & Ho, 1997; O’Brien, 2008) Novelty (Webster & Ho, 1997; O’Brien, 2008) Reputation, trust and expectation (Attfield et al, 2011) Richness and control Positive Affect (O’Brien & Toms, 2008) (O’Brien, Lalmas & Yom-Tov, 2013) (Jacques et al, 1995; Webster & Ho, 1997)
  • 7. Measuring user engagement Measures   Characteristics   SelfQuestionnaire, interview, report, Subjective reported product reaction cards, think-aloud Short- and long-term engagement Lab and field Small-scale Cognitive Task-based methods (time spent, engagement follow-on task) Objective Short-term Lab and field Physiological measures (e.g. EEG, Small-scale and largeSCL, fMRI, eye tracking, mousescale tracking) Interaction Web analytics engagement metrics + models Objective Short- and long-term Field Large-scale
  • 8. User engagement in web search – around “returning relevant results to the users” (Slaney, 2011) satisfying user information needs §  Clickthrough rate (CTR) §  Dwell time (on landing page) §  Time to first click §  Skipping §  Multimedia search activities often driven by entertainment needs, not by information needs. §  Displaying rich information on result pages (restaurant phone number) means that users do not need to click. §  Abandonment rate §  Number of query reformulations §  Search engine switching §  Cumulative gain family of metrics, precision at rank k, …
  • 9. This talk What is user engagement? What are the characteristics of user engagement? How to measure user engagement? What is user engagement in web search? What is an engaging click? 1.  inter-session 2.  online multi-tasking 3.  downstream engagement 4.  serendipity work on user engagement across web applications implications to web search
  • 10. (Dupret & Lalmas, 2013) •  •  Domain: Yahoo Answers Japan Study: Inter-session engagement metric If users find a web application interesting, engaging or useful, they will return to it sooner.
  • 11. Absence time and survival analysis survival Analysis: high hazard rate = short absence short absence is a sign of loyalty important indication of user engagement
  • 12. Using absence time to compare 6 ranking functions (buckets) on Yahoo Answers Japan 1.  Returning relevant results is important, but is not enough to keep returning to the search application 2.  Clicks after the 5th results reflect poorer user experience; users cannot find what they are looking for Endurability 3.  No click means a bad user experience 4.  Clicking lower in the ranking suggests more careful choice from the user 5.  Clicking at bottom is a sign of low quality overall ranking 6.  Users finding their answers quickly (click sooner) return sooner to the search application 7.  Returning to the same search result page is a worse user experience than reformulating the query.
  • 13. (Lehmann et al, 2013) •  •  Domain: 700+ web applications Study: Online multi-tasking Online multi-tasking affects the way users interact (or engage) with sites.
  • 14. Online multi-tasking – and search 181K users, 2 months browser data, 600 sites, 4.8M sessions • only 40% of the sessions have no site revisitation •  commonly accessed sites between visits à search 22%, navigation 12%, social 8% •  for some sites (e-commerce) same sites are accessed between visits à one task? •  no patterns for sites such as mail, social à anchor, habit? •  longer time between visits à a different task (new search) •  more vs less times spent at each revisit à increased vs shift of attention
  • 15. Revisitation patterns average attention mail sites [decreasing attention] search sites [increasing attention] auction sites [complex attention] 100% 54% 36% 26% 20% 17% 14% 12% 10% 100% 62% 41% 29% 21% 16% 13% 10% 8% 100% 69% 54% 44% 38% 33% 29% 26% 23% 100% 67% 54% 46% 41% 35% 31% 29% 26% p-value < 0.05 p-value < 0.05 m = -0.288 p-value < 0.05 m = 0.063 p-value = 0.24 m = 0.142 ● 0.8 ● 12 11 11.2 0.4 ● ● ● ● ● ● 10 ● ● 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 0.4 ● ● 1 2 3 4 5 6 7 8 9 0.8 ● ● ● 11.0 ● Motivation, interests, incentives, and benefits k [kth visit on site] 0.0 0.0 §  10% users accessed a site 9+ times (23% for search sites); 28% at least four 1 2 3 4 5 6 7 8 9 1 2 3 4 1 times (44%7for search sites) 5 6 7 8 9 2 3 4 5 6 8 9 1 2 3 4 5 6 7 8 9 0.0 0.0 ● §  48% sites visited at least 9 times §  Revisitation “level” depends on site 0.4 0.8 1 2 3 4 5 6 7 8 9 % of navigation type ● ● ● ● 0.4 ● ● ● 10.8 12 ● ● ● ● ● 0.8 ● ● 11 11.2 ● ● 13 ● ● 10 ● 10.8 % of total page views on site proportion of users k [kth visit on site] k [kth visit on site] k [kth visit on site] Hyperlinking §  Activity on site decreases with each Teleporting but activity on many search revisit Backpaging sites increases
  • 16. (Yom-Tov et al, 2013) •  •  Domain: Network of sites Study: Downstream engagement Success of a site depends on itself, but also on how it is reached from other sites.
  • 17. Downstream user engagement: engagement across a network of sites Large online providers (AOL, Google, Yahoo!, MSN, etc.) offer not one service (site), but a network of sites Downstream engagement for site A (% remaining session time) Site A Provider sites User session
  • 18. Influential features (50 Yahoo sites, 250K+ users, 1.9M sessions) o Time of day o Number of (non-image/non-video) links to Yahoo! sites in HTML body o Average rank of Yahoo! links on page o Number of (non-image/non-video) links to non-Yahoo! sites in HTML body o Number of span tags (tags that allow adding style to content or manipulating content, e.g. JavaScript) o  Link placements and number of Yahoo links can influence downstream engagement o  Not new, but here shown to hold also across sites Richness and control o  Links to non-Yahoo sites have a positive effect on downstream engagement o  Possibly because when users are faced with abundance of outside links they decide to focus their attention on a central content provider, rather than visiting multitude of external sites
  • 19. (Bordino, Mejova & Lalmas, 2013) •  •  Domain: social media (Yahoo! Answers and Wikipedia) Study: serendipity (in entity search) Interesting search results may promote serendipitous browsing.
  • 20. Yahoo! Answers community-driven question & answer portal §  67 336 144 questions & 261 minimally curated 770 047 answers opinions, gossip, personal info §  January 1, 2010 –of view variety of points December 31, 2011 §  English-language Entity Search vs Wikipedia community-driven encyclopedia •  3 795 865 articles curated •  as of endknowledge high-quality of December 2011 variety of niche topics •  English Wikipedia we build an entity-driven serendipitous search system based on entity networks extracted from Wikipedia and Yahoo! Answers Serendipity finding something good or useful while not specifically looking for it, serendipitous search systems provide relevant and interesting results
  • 21. retrieve entities most related to a query entity using random walk Wikipedia Yahoo! Answers •  •  Annotator agreement (overlap): 0.85 Average overlap in top 5 results: <1
  • 22. Serendipity “making fortunate discoveries by accident” Serendipity = unexpectedness + relevance “Expected” result baselines from web search Baseline   Top:  5  en&&es  that  occur  most  frequently   in  top  5  search  from  Bing  and  Google   Top  –WP:  same  as  above,  but  excluding     Wikipedia  page  from  results   Rel:  top  5  en&&es  in  the  related  query     sugges&ons  provided  by  Bing  and  Google   Rel  +  Top:  union  of  Top  and  Rel   | relevant & unexpected | / | unexpected | number of serendipitous results out of all of the unexpected results retrieved Data   WP   YA   WP   YA   WP   YA   WP   YA   0.63  (0.58)   0.69  (0.63)   0.63  (0.58)   0.70  (0.64)   0.64  (0.61)   0.70  (0.65)   0.61  (0.54)   0.68  (0.57)   | relevant & unexpected | / | retrieved | serendipitous out of all retrieved
  • 23. Interestingness ≠ Relevance Interesting > Relevant Oil Spill à Robert Pattinson à Egypt à Penguins in Sweaters WP Water for Elephants WP Ptolemaic Kingdom WP & YA Relevant > Interesting Lady Gaga à Britney Spears Novelty WP Egypt à Cairo Conference WP Netflix à Blu-ray Disc YA
  • 24. Take-away message §  Search is not just about specific information needs §  People search for many other reasons ›  Navigation ›  ›  Transaction Fun (ECIR 2012 workshop) ›  Etc. §  Engagement in search is to view search activities as part of the current overall task of a user §  We never know what we get if we are ready to explore ›  Users do things that no one expects, not even them! ›  (like staying inside Yahoo! in spite of having many links to go elsewhere) So a link is not everything, for search too! §  Summarizing, in search we need to look at user engagement in a broader way
  • 25. Thank you Acknowledgements: Ricardo Baeza-Yates, Ilaria Bordino, George Dupret, Janette Lehmann, Yelena Mejova and Elad Yom-Tov. Blog: labtomarket.wordpress.com A first version of this talk was given by Ricardo Baeza-Yates, SIGIR 2013 Industry Day