Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Detecting Good Abandonment in Mobile Search

1,363 views

Published on

Web search queries for which there are no clicks are referred to as abandoned queries and are usually considered
as leading to user dissatisfaction. However, there are many
cases where a user may not click on any search result page
(SERP) but still be satis ed. This scenario is referred to
as good abandonment and presents a challenge for most ap-
proaches measuring search satisfaction, which are usually
based on clicks and dwell time. The problem is exacerbated
further on mobile devices where search providers try to in-
crease the likelihood of users being satis ed directly by the
SERP. This paper proposes a solution to this problem us-
ing gesture interactions, such as reading times and touch
actions, as signals for di erentiating between good and bad
abandonment. These signals go beyond clicks and charac-
terize user behavior in cases where clicks are not needed to
achieve satisfaction. We study di fferent good abandonment
scenarios and investigate the di erent elements on a SERP
that may lead to good abandonment. We also present an
analysis of the correlation between user gesture features and
satisfaction. Finally, we use this analysis to build models to
automatically identify good abandonment in mobile search
achieving an accuracy of 75%, which is signi ficantly better
than considering query and session signals alone. Our fundings have implications for the study and application of user
satisfaction in search systems.

Published in: Internet
  • Be the first to comment

Detecting Good Abandonment in Mobile Search

  1. 1. Detecting Good Abandonment in Mobile Search Kyle Williams Julia Kiseleva Aidan C. Crook Imed Zitouni Ahmed Hassan Awadallah Madian Khabsa Pennsylvania State University Eindhoven University of Technology Microsoft WWW’16, Montréal, Québec, Canada
  2. 2. Mobile Search
  3. 3. Mobile Search • More and more popular: 2008  31% 2013  63% • Mobile Search differs from traditional search [Human et. al, 2009] • On Mobiles users are satisfied by the SERP [Li et. al, 2009] • Mobiles screen is much smaller • Mobiles are used on the way
  4. 4. Mobile Search • More and more popular: 2008  31% 2013  63% • Mobile Search differs from traditional search [Human et. al, 2009] • On Mobiles users are satisfied by the SERP [Li et. al, 2009] • Mobiles screen is much smaller • Mobiles are used on the way Search Engines need to adapt And to Evaluate!
  5. 5. Knowledge Pane Image Answer
  6. 6. Knowledge Pane Image Answer Image Answer Organic Results: Snippets
  7. 7. Knowledge Pane Image Answer Image Answer Organic Results: Snippets Knowledge Pane
  8. 8. Evaluating User Satisfaction • We need metrics to evaluate user satisfaction • Good abandonment [Human et. al, 2009]: Mobile: 36% of abandoned queries in were likely good Desktop: 14.3% • Traditional methods use implicit signals: clicks and dwell time
  9. 9. Evaluating User Satisfaction • We need metrics to evaluate user satisfaction • Good abandonment [Human et. al, 2009]: Mobile: 36% of abandoned queries in were likely good Desktop: 14.3% • Traditional methods use implicit signals: clicks and dwell time Don’t work
  10. 10. Our Main Research Problem In the absence of clicks, what is the relationship between a user's gestures and satisfaction and can we use gestures to detect satisfaction and good abandonment?
  11. 11. Research Questions • RQ1: What SERP elements are the sources of good abandonment in mobile search? • RQ2: Do a user's gestures provide signals that can be used to detect satisfaction and good abandonment in mobile search? • RQ3: Which user gestures provide the strongest signals for satisfaction and good abandonment?
  12. 12. Research Questions • RQ1: What SERP elements are the sources of good abandonment in mobile search? • RQ2: Do a user's gestures provide signals that can be used to detect satisfaction and good abandonment in mobile search? • RQ3: Which user gestures provide the strongest signals for satisfaction and good abandonment? USERSTUDY
  13. 13. Research Questions • RQ1: What SERP elements are the sources of good abandonment in mobile search? • RQ2: Do a user's gestures provide signals that can be used to detect satisfaction and good abandonment in mobile search? • RQ3: Which user gestures provide the strongest signals for satisfaction and good abandonment? USERSTUDY CROWDSOURCING
  14. 14. User Study Participants 75% 25% GENDER Male Female 55% 45% LANGUAGE English Other 82% 8% 2% 8% EDUCATION Computer Science Electrical Engineering Mathematics Other • 60 Participants • 25.53 +/- 5.42 years
  15. 15. User Study Design • Video Instructions (same for all participants) • Tasks: 1. A conversion between the imperial and metric systems 2. Determining if it was a good time to phone a friend in another part of the world 3. Finding the score from a recent game of the user’s favorite sports team 4. Finding the user's favorite celebrity's hair color 5. Finding the CEO of a company that lost most of its value in the last 10 years
  16. 16. Find out what is the hair color of your favorite celebrity
  17. 17. Questionnaire • Were you able to complete the task? o Yes/No • Where did you find the answer? o Answer Box, Image, SERP, Visited Website • Which query led you to finding the answer? o First, Second, Third, >= Fourth • How satisfied are you with your experience in this task? o 5-point Likert scale • Did you put in a lot of effort to complete the task? o 5-point Likert scale
  18. 18. Questionnaire • Were you able to complete the task? o Yes/No • Where did you find the answer? o Answer Box, Image, SERP, Visited Website • Which query led you to finding the answer? o First, Second, Third, >= Fourth • How satisfied are you with your experience in this task? o 5-point Likert scale • Did you put in a lot of effort to complete the task? o 5-point Likert scale 5 Tasks ~20 Minutes
  19. 19. User Study Data • Total queries – 607  563 • Abandoned queries – 576  461 • Potential abandonment tasks – 274
  20. 20. User Study Data • Total queries – 607  563 • Abandoned queries – 576  461 • Potential abandonment tasks – 274 Binary Labels
  21. 21. Crowdsourcing Procedure Random sample of abandoned queries from the search logs of a personal digital assistant during one week in June 2015 (no query suggestion)
  22. 22. Crowdsourcing Procedure Query: Peniston Previous Query: third eroics
  23. 23. Crowdsourcing Data • Total amount of queries – 3,895 • Judgments agreement (3 per one query) – 73% • After filtering: SAT – 1,565 and DSAT – 1,924
  24. 24. RQ1: Reasons of Good Abandonment
  25. 25. RQ1: Reasons of Good Abandonment Mean of Satisfaction
  26. 26. Query and Session Features • Session duration • Number of queries in session Session Features
  27. 27. Query and Session Features • Session duration • Number of queries in session • Index of query within session • Time to next query • Query length (number of words) • Is this query a reformulation • Was this query reformulated Session Features Query Features
  28. 28. Query and Session Features • Session duration • Number of queries in session • Index of query within session • Time to next query • Query length (number of words) • Is this query a reformulation • Was this query reformulated • Click count • Number of SAT clicks (> 30 sec) • Number of back-click clicks (< 30 sec) Session Features Query Features Click Features
  29. 29. Baseline 1:Click & Dwell • Session duration • Number of queries in session • Index of query within session • Time to next query • Query length (number of words) • Is this query a reformulation • Was this query reformulated • Click count • Number of SAT clicks (> 30 sec) • Number of back-click clicks (< 30 sec) Session Features Query Features Click Features Click > 30 sec No Refomul ation B1:Click,Dwellwith noReformulation
  30. 30. Baseline 2: Optimistic • Session duration • Number of queries in session • Index of query within session • Time to next query • Query length (number of words) • Is this query a reformulation • Was this query reformulated • Click count • Number of SAT clicks (> 30 sec) • Number of back-click clicks (< 30 sec) Session Features Query Features Click Features NO Click NO Refomul ation B2:Optimistic
  31. 31. Baseline 3: Query-Session Model • Session duration • Number of queries in session • Index of query within session • Time to next query • Query length (number of words) • Is this query a reformulation • Was this query reformulated • Click count • Number of SAT clicks (> 30 sec) • Number of back-click clicks (< 30 sec) Session Features Query Features Click Features B3:Query-SessionModel: TrainingRandomForest
  32. 32. Gesture Features (1) • Viewport features swipes-related: o up swipes and down swipes o changes in swipe direction o swiped distance in pixels and average swiped distance o swipe distance divided by time spent on the SERP
  33. 33. Gesture Features (1) • Viewport features swipes-related: o up swipes and down swipes o changes in swipe direction o swiped distance in pixels and average swiped distance o swipe distance divided by time spent on the SERP • Time To Focus o Time to focus on Answer o Time to Focus on Organic Search Results
  34. 34. 3 seconds 6 seconds 33% of ViewPort 66% of ViewPort ViewPortHeight 2 seconds 20% of ViewPort 1s 4s 0.4s 5.4s+ + = GF(2): Attributed Reading Time
  35. 35. 400 pixels 300 pixels Attributed Reading Time: 5.4s Pixel Area: (400 pix x 300 pix) 0.045 ms/pix2= GF (3): Attributed Reading Time Per Pixel
  36. 36. Models: Detecting Good Abandonment M1: Gesture Model: Training Random Forest based on gesture features M2: Gesture Model + Query and Session Features: Training Random Forest based on gesture, query and session features
  37. 37. RQ2: Are gestures useful? (1) On only abandoned user study data: 148 SAT queries and 313 DSAT queries
  38. 38. RQ2: Are gestures useful? (2) On crowdsourced data: 1565 SAT queries and 1924 DSAT queries
  39. 39. RQ2: Are gestures useful? (3) On all user study data: 179 SAT queries and 384 DSAT queries Gestures Features are useful to detect user satisfaction in general!
  40. 40. Conclusions • RQ1: What SERP elements are the sources of good abandonment in mobile search? Answer, Images and Snippet • RQ2: Do a user's gestures provide signals that can be used to detect satisfaction and good abandonment in mobile search? Yes • RQ3: Which user gestures provide the strongest signals for satisfaction and good abandonment Time spent interacting with Answers is positively correlated. Swipe actions and time spent with SERP is negatively correlated
  41. 41. • Answer, Images and Snippet are potentially source of the good abandonment • User gestures provide useful signals to detect good abandonment • Time spent interacting with Answers is positively correlated. Swipe actions and time spent with SERP is negatively correlated Questions?

×