Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Modeling and Detecting
Changes in User Satisfaction
Julia Kiseleva*, Eric Crestan, Riccardo Brigo, Roland Dittel
*Eindhove...
Want to go to CIKM
conference
QUERY SERP
What is User Satisfaction?
What is User Satisfaction?
QUERY SERP
,
What is User Satisfaction?
QUERY SERP
,
What is User Satisfaction?
QUERY SERP
,Pr (Ref.)
Assumption: If a “significant” amount of users
reformulate a query with a...
World May Change User Preferences
QUERY SERP
,
QUERY SERP
ti
ti+1
,
Timeline
Pr ti =
Pr ti+1 =
How Can We Detect the Changes?
QUERY SERP
,
QUERY SERP
ti
ti+1
,
| Pr ti - Pr ti+1 |
Timeline
Pr ti =
Pr ti+1 =
How Can We Detect the Changes?
• There are many definitions in the literature
• We use the query expansion
o new years wallpaper IS REFORMULATED WITH 201...
An Example of the Drift in
Reformulation Signal
The Explanation of the Drift
Before November 2013 After November 2013
The Question:
“How to detect
this kind of
changes?”
• Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can
change ...
• Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change ...
• Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change ...
• Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change ...
• Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change ...
• Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change ...
• Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can
change ...
Detecting Drifts in
Reformulation Signal
Query: “cikm conference”
0.1
TimeLinet0
0.1 0.2 0.2 0.3
Reformulation: “2014”
Win...
Detecting Drifts in
Reformulation Signal
Query: “cikm conference”
0.1
TimeLinet0 ti+ t
0.1 0.2 0.2 0.3 0.7 0.8 0.8
Reformu...
Calculating Threshold eout
Confidence
Variance at W = W0 U W1
m = 1/(1/n0 + 1/n1)
eout
Learn
reformulation
model M
User Behavior
Logs
t0 Timelineti+
Learn
reformulation
model M
User Behavior
Logs
t0
Incoming User
Behavior logs
Timeline
Detect changes in model M
If change...
Learn
reformulation
model M
User Behavior
Logs
ti
Incoming User
Behavior logs
Timeline
Detect changes in model M
If change...
Learn
reformulation
model M
User Behavior
Logs
t0
Incoming User
Behavior Logs
Timeline
Detect changes in model M
If change...
o Dataset consists of 6 months
of the behavioral log data
from a commercial search
engine
o The training window size is
on...
Evaluation
Results
o We successfully leveraged the concept drift detection
techniques to detect changes in user satisfaction
o The proposed t...
Questions?
Questions?
o We successfully leveraged the concept drift detection
techniques
o The proposed technique works in unsupervised way
o La...
Upcoming SlideShare
Loading in …5
×

Modelling and Detecting Changes in User Satisfaction

699 views

Published on

modelling and detecting changes in user satisfaction
CIKM 2014

Published in: Data & Analytics
  • Be the first to comment

Modelling and Detecting Changes in User Satisfaction

  1. 1. Modeling and Detecting Changes in User Satisfaction Julia Kiseleva*, Eric Crestan, Riccardo Brigo, Roland Dittel *Eindhoven University of Technology Microsoft Bing
  2. 2. Want to go to CIKM conference QUERY SERP What is User Satisfaction?
  3. 3. What is User Satisfaction? QUERY SERP ,
  4. 4. What is User Satisfaction? QUERY SERP ,
  5. 5. What is User Satisfaction? QUERY SERP ,Pr (Ref.) Assumption: If a “significant” amount of users reformulate a query with a particular SERP it is an indication of changing in user preferences
  6. 6. World May Change User Preferences
  7. 7. QUERY SERP , QUERY SERP ti ti+1 , Timeline Pr ti = Pr ti+1 = How Can We Detect the Changes?
  8. 8. QUERY SERP , QUERY SERP ti ti+1 , | Pr ti - Pr ti+1 | Timeline Pr ti = Pr ti+1 = How Can We Detect the Changes?
  9. 9. • There are many definitions in the literature • We use the query expansion o new years wallpaper IS REFORMULATED WITH 2014 o medals Olympics IS REFORMULATED WITH 2014 o ct 40ez IS REFORMULATED WITH 2013 o march 31 holiday IS REFORMULATED WITH 2014 o … Detecting Query Reformulation
  10. 10. An Example of the Drift in Reformulation Signal
  11. 11. The Explanation of the Drift Before November 2013 After November 2013 The Question: “How to detect this kind of changes?”
  12. 12. • Change detection techniques o In dynamically changing and non-stationary environments, the data distribution can change over time yielding the phenomenon of concept drift o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable) given the input (input features) • Concept drift types: Change Detection Techniques
  13. 13. • Change detection techniques o In dynamically changing and non-stationary environments, the data distribution can change over time yielding the phenomenon of concept drift o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable) given the input (input features) • Concept drift types: Time Datamean Sudden/abrupt Disambiguation such as “flawless Beyoncé” Change Detection Techniques
  14. 14. • Change detection techniques o In dynamically changing and non-stationary environments, the data distribution can change over time yielding the phenomenon of concept drift o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable) given the input (input features) • Concept drift types: Time Datamean Incremental Disambiguation such as “cikm conference 2014” Change Detection Techniques
  15. 15. • Change detection techniques o In dynamically changing and non-stationary environments, the data distribution can change over time yielding the phenomenon of concept drift o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable) given the input (input features) • Concept drift types: Time Datamean Gradual Breaking news such as “idaho bus crash investigation” Change Detection Techniques
  16. 16. • Change detection techniques o In dynamically changing and non-stationary environments, the data distribution can change over time yielding the phenomenon of concept drift o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable) given the input (input features) • Concept drift types: Time Datamean Reoccurring Seasonal change such as “black Friday 2014” Change Detection Techniques
  17. 17. • Change detection techniques o In dynamically changing and non-stationary environments, the data distribution can change over time yielding the phenomenon of concept drift o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable) given the input (input features) • Concept drift types: Time Datamean Change Detection Techniques
  18. 18. • Change detection techniques o In dynamically changing and non-stationary environments, the data distribution can change over time yielding the phenomenon of concept drift o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable) given the input (input features) • Concept drift types: Time Datamean Sudden/abru pt Incremental Gradual Reoccurring concepts Outlier (not concept drift) Disambiguation such as “medal olympics 2014” Seasonal change such as “black Friday 2014” Breaking news such as “idaho bus crash investigation” Disambiguation such as “cikm conference 2014” Change Detection Techniques
  19. 19. Detecting Drifts in Reformulation Signal Query: “cikm conference” 0.1 TimeLinet0 0.1 0.2 0.2 0.3 Reformulation: “2014” Window W0 ti
  20. 20. Detecting Drifts in Reformulation Signal Query: “cikm conference” 0.1 TimeLinet0 ti+ t 0.1 0.2 0.2 0.3 0.7 0.8 0.8 Reformulation: “2014” Window W0 Window W1 ti E(W0) E(W1) Size of Window W1 = n1Size of Window W0 = n0 The upcoming conference event If |E(W1) - E(W2)|> eout Then Drift Detected
  21. 21. Calculating Threshold eout Confidence Variance at W = W0 U W1 m = 1/(1/n0 + 1/n1) eout
  22. 22. Learn reformulation model M User Behavior Logs t0 Timelineti+
  23. 23. Learn reformulation model M User Behavior Logs t0 Incoming User Behavior logs Timeline Detect changes in model M If change detected else Do Nothing ti ti+ t
  24. 24. Learn reformulation model M User Behavior Logs ti Incoming User Behavior logs Timeline Detect changes in model M If change detected else Do Nothing ti+w1 ti+w1+w2 Alarm: Change of user satisfaction detected for pairs : {<Qi, SERPi>}1<i<n
  25. 25. Learn reformulation model M User Behavior Logs t0 Incoming User Behavior Logs Timeline Detect changes in model M If change detected else Do Nothing ti ti+ t 1) List of reformulation terms per query 2) List of URLs per reformulation Alarm: Change of user satisfaction detected for pairs : {<Qi, SERPi>}1<i<n
  26. 26. o Dataset consists of 6 months of the behavioral log data from a commercial search engine o The training window size is one month o The test window size is two weeks Experimentation
  27. 27. Evaluation
  28. 28. Results
  29. 29. o We successfully leveraged the concept drift detection techniques to detect changes in user satisfaction o The proposed technique works in unsupervised way o Large scale evaluation has been performed o Classification of the drift type is needed o Prediction of the lifetime of the drift would help Conclusion and Future Work
  30. 30. Questions?
  31. 31. Questions?
  32. 32. o We successfully leveraged the concept drift detection techniques o The proposed technique works in unsupervised way o Large scale evaluation has been performed o Classification of the drift type is needed o Prediction of the lifetime of the drift would help Conclusion and Future Work

×