Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Behavioral Dynamics from the SERP’s Perspective: What are Failed SERPs and How to Fix Them?

669 views

Published on

Web search is always in a state of flux: queries, their intent, and the
most relevant content are changing over time, in predictable and unpredictable
ways. Modern search technology has made great strides
in keeping up to pace with these changes, but there remain cases of
failure where the organic search results on the search engine result
page (SERP) are outdated, and no relevant result is displayed.
Failing SERPs due to temporal drift are one of the greatest frustrations
of web searchers, leading to search abandonment or even
search engine switch. Detecting failed SERPs timely and providing
access to the desired out-of-SERP results has huge potential to improve
user satisfaction. Our main findings are threefold: First, we
refine the conceptual model of behavioral dynamics on the web by
including the SERP and defining (un)successful SERPs in terms of
observable behavior. Second, we analyse typical patterns of temporal
change and propose models to predict query drift beyond the
current SERP, and ways to adapt the SERP to include the desired
results. Third, we conduct extensive experiments on real world
search engine traffic demonstrating the viability of our approach.
Our analysis of behavioral dynamics at the SERP level gives new
insight in one of the primary causes of search failure due to temporal
query intent drifts. Our overall conclusion is that the most
detrimental cases in terms of (lack of) user satisfaction lead to the
largest changes in information seeking behavior, and hence to observable
changes in behavior we can exploit to detect failure, and
moreover not only detect them but also resolve them.

Published in: Internet
  • Be the first to comment

Behavioral Dynamics from the SERP’s Perspective: What are Failed SERPs and How to Fix Them?

  1. 1. Behavioral Dynamics from the SERP’s Perspective: What are Failed SERPs and How to Fix Them? Julia Kiseleva, Jaap Kamps, Vadim Nikulin , Nikita Makarov Eindhoven University of Technology University of Amsterdam Yandex CIKM’15, Melbourne, Australia
  2. 2. Want to go to CIKM conference QUERY SERP What is User Satisfaction?
  3. 3. Want to go to CIKM conference QUERY SERP What is User Satisfaction? ? A YEAR AGO
  4. 4. Want to go to CIKM conference QUERY SERP What is User Satisfaction? ? TODAY
  5. 5. Want to go to CIKM conference QUERY SERP What is User Satisfaction? What about freshness
  6. 6. Want to go to CIKM conference QUERY SERP What is User Satisfaction? What about freshness ???
  7. 7. What is User Satisfaction? QUERY SERP ,
  8. 8. What is User Satisfaction? QUERY SERP ,
  9. 9. Changes in User Intents WikipediaPageView 2015
  10. 10. Changes in User Intents WikipediaPageView 2015 Malaysia Airlines Flight 370
  11. 11. Changes in User Intents WikipediaPageView 2015 Malaysia Airlines Flight 370
  12. 12. Changes in User Intents WikipediaPageView 2015 Malaysia Airlines Flight 370 Malaysia Airlines Flight 17
  13. 13. • By analyzing behavioral dynamics at the SERP level, • can we detect an important class of detrimental cases (such as search failure) • based on changes in observable behavior caused by low user satisfaction? Research Problem
  14. 14. ti ti+1 Timeline How Can We Detect the Changes? QUERY SERP , QUERY SERP ,
  15. 15. ti ti+1 Timeline How Can We Detect the Changes? QUERY SERP , QUERY SERP , SAT DSAT
  16. 16. ti ti+1 Timeline How Can We Detect the Changes? BF1 = Reformulation Signal (RS) BF2 = Abandonment Signal (AS) BF3 = Volume Signal (VS) QUERY SERP , QUERY SERP , BF4 = Click Position Signal (CS)
  17. 17. ti ti+1 Timeline How Can We Detect the Changes? BF1 = Reformulation Signal (RS) BF2 = Abandonment Signal (AS) BF3 = Volume Signal (VS) QUERY SERP , QUERY SERP , BF4 = Click Position Signal (CS)
  18. 18. • Change detection techniques o In dynamically changing and non-stationary environments, the data distribution can change over time yielding the phenomenon of concept drift o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable) given the input (input features) • Concept drift types: Change Detection Techniques
  19. 19. • Change detection techniques o In dynamically changing and non-stationary environments, the data distribution can change over time yielding the phenomenon of concept drift o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable) given the input (input features) • Concept drift types: Time Datamean Sudden/abru pt Incremental Gradual Reoccurring Outlier (not concept drift) The example: “flawless Beyoncé” Seasonal change such as “black Friday 2014” The example: “idaho bus crash investigation” The example: “cikm conference 2015” Change Detection Techniques
  20. 20. Detecting Drift in User Satisfaction QUERY SERP ,BFi Pti BFi ( ),
  21. 21. Detecting Drift in User Satisfaction QUERY SERP ,BFi Pti BFi ( ) QUERY SERP ,Pti+1 BFi ( ) , ,
  22. 22. Detecting Drift in User Satisfaction QUERY SERP ,BFi Pti BFi ( ) QUERY SERP ,Pti+1 BFi ( ) , , Failed SERP !!!
  23. 23. Failed SERP
  24. 24. Failed SERP schedule of matches of the World Cup 2014 schedule of matches of the World Cup 2014 ON TV
  25. 25. Detecting Drifts in Behavioral Signals Query: “cikm conference” 0.1 TimeLinet0 0.1 0.2 0.2 0.3 Reformulation: “2015” Window W0 ti
  26. 26. Query: “cikm conference” 0.1 TimeLinet0 ti+ t 0.1 0.2 0.2 0.3 0.7 0.8 0.8 Reformulation: “2015” Window W0 Window W1 ti E(W0) E(W1) Size of Window W1 = n1Size of Window W0 = n0 Drift If |E(W1) - E(W2)|> eout Then Drift Detected Detecting Drifts in Behavioral Signals
  27. 27. Calculating Threshold eout Confidence Variance at W = W0 U W1 m = 1/(1/n0 + 1/n1) eout
  28. 28. Time Datamean Sudden drift ti ti+1 ti+1+W1 Sudden VS Incremental Drifts
  29. 29. Time Datamean Sudden drift Sudden Drift: Disambiguation such as ‘medal Olympics 2016’ ti ti+1 ti+1+W1 Sudden VS Incremental Drifts
  30. 30. Time Datamean Sudden drift Sudden Drift: Disambiguation such as ‘medal Olympics 2016’ ti ti+1 ti+1+W1 Sudden VS Incremental Drifts Abandonment Signal (AS) Volume Signal (VS)
  31. 31. Time Datamean Sudden drift Incremental Drift Sudden Drift: Disambiguation such as ‘medal Olympics 2016’ Incremental Drift: Disambiguation such as ‘CIKM conference 2015’ ti ti+1 ti+1+W1 tj tj+1 tj+1+W2 Sudden VS Incremental Drifts
  32. 32. Time Datamean Sudden drift Incremental Drift Sudden Drift: Disambiguation such as ‘medal Olympics 2016’ Incremental Drift: Disambiguation such as ‘CIKM conference 2015’ ti ti+1 ti+1+W1 tj tj+1 tj+1+W2 Sudden VS Incremental Drifts Click Position Signal (CS) Abandonment Signal (AS) Volume Signal (VS)
  33. 33. Time Datamean Sudden drift Incremental Drift Sudden Drift: Disambiguation such as ‘medal Olympics 2016’ Incremental Drift: Disambiguation such as ‘CIKM conference 2015’ ti ti+1 ti+1+W1 tj tj+1 tj+1+W2 W1 << W2 Sudden VS Incremental Drifts Click Position Signal (CS) Abandonment Signal (AS) Volume Signal (VS)
  34. 34. Time Datamean Reoccurring drift Disambiguation such as ‘movie premieres November 2014’ Disambiguation such as ‘movie premieres December 2014’ Disambiguation such as ‘movie premieres January 2015’ + _ Changesinqueryintent _ _ + + Positive Sudden Drift Negative Sudden Drift Negative Sudden Drift Negative Sudden Drift Positive Sudden Drift Positive Sudden Drift Reoccurring Drifts If sign(E(Wi+1) - E(Wi)) > 0 then “+” If sign(E(Wi+1) - E(Wi)) < 0 then “-”
  35. 35. Gradual Drifts Time Datamean Gradual drift Disambiguation such as ‘novak djokovic fiancée’ Disambiguation such as ‘novak djokovic wedding’ Disambiguation such as ‘novak djokovic baby’ + + +__ _ Positive Sudden Drift Negative Sudden Drift Positive Incremental Drift Changesinqueryintent
  36. 36. o Dataset consists of 12 months of the behavioral log data from Yandex (2015) o ~25 millions users per day o In total ~ 150 millions of queries per day o Train Period – one month o Test Period – 3, 7, and 14 days Experimentation
  37. 37. FrequencyofdetectedDrifts
  38. 38. o 100s of thousands of query drifts detected • huge number, but small fraction of traffic o Over 200,000 unique <Q,Q’> pairs o Revisions are varied • unique revision term(s) occurring in 3-4 unique pairs • 2-3 % are year revisions (‘2014’, ‘2015’) • 17-18 % of revisions contain any number o Detect far more revisions than standard rules/templates • Queries and revision in many language • Demonstrates general applicability of the approach Detected Query Drifts
  39. 39. Evaluation Return most clicked URL
  40. 40. Results
  41. 41. Results Incremental Drift
  42. 42. • We conducted a conceptual analysis of success and failure at the SERP level o we introduced the concept of a successful and failed SERP o we analyzed their behavioral consequences identifying indicators of success and failure • We conducted an analysis of different types of drifts in query intent over time o we studied different changes in query intent: sudden, incremental, gradual and reoccurring o we introduced an unsupervised approach to detect failed SERPs caused by drift (sudden, incremental) • We tested our detector on massive raw search logs Conclusions
  43. 43. Questions?
  44. 44. • We conducted a conceptual analysis of success and failure at the SERP level o we introduced the concept of a successful and failed SERP o we analyzed their behavioral consequences identifying indicators of success and failure • We conducted an analysis of different types of drifts in query intent over time o we studied different changes in query intent: sudden, incremental, gradual and reoccurring o we introduced an unsupervised approach to detect failed SERPs caused by drift (sudden, incremental) • We tested our detector on massive raw search logs Conclusions

×