Modelling and Detecting Changes in User Satisfaction
1. Modeling and Detecting
Changes in User Satisfaction
Julia Kiseleva*, Eric Crestan, Riccardo Brigo, Roland Dittel
*Eindhoven University of Technology
Microsoft Bing
2. Want to go to CIKM
conference
QUERY SERP
What is User Satisfaction?
5. What is User Satisfaction?
QUERY SERP
,Pr (Ref.)
Assumption: If a “significant” amount of users
reformulate a query with a particular SERP it is an
indication of changing in user preferences
9. • There are many definitions in the literature
• We use the query expansion
o new years wallpaper IS REFORMULATED WITH 2014
o medals Olympics IS REFORMULATED WITH 2014
o ct 40ez IS REFORMULATED WITH 2013
o march 31 holiday IS REFORMULATED WITH 2014
o …
Detecting Query Reformulation
11. The Explanation of the Drift
Before November 2013 After November 2013
The Question:
“How to detect
this kind of
changes?”
12. • Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can
change over time yielding the phenomenon of concept drift
o The real concept drift refers to changes in the conditional distribution of the output
(i.e., target variable) given the input (input features)
• Concept drift types:
Change Detection Techniques
13. • Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change over time
yielding the phenomenon of concept drift
o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable)
given the input (input features)
• Concept drift types:
Time
Datamean
Sudden/abrupt
Disambiguation
such as
“flawless Beyoncé”
Change Detection Techniques
14. • Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change over time
yielding the phenomenon of concept drift
o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable)
given the input (input features)
• Concept drift types:
Time
Datamean
Incremental
Disambiguation
such as
“cikm conference
2014”
Change Detection Techniques
15. • Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change over time
yielding the phenomenon of concept drift
o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable)
given the input (input features)
• Concept drift types:
Time
Datamean
Gradual
Breaking news
such as
“idaho bus crash
investigation”
Change Detection Techniques
16. • Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change over time
yielding the phenomenon of concept drift
o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable)
given the input (input features)
• Concept drift types:
Time
Datamean
Reoccurring
Seasonal change
such as
“black Friday 2014”
Change Detection Techniques
17. • Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can change over time
yielding the phenomenon of concept drift
o The real concept drift refers to changes in the conditional distribution of the output (i.e., target variable)
given the input (input features)
• Concept drift types:
Time
Datamean
Change Detection Techniques
18. • Change detection techniques
o In dynamically changing and non-stationary environments, the data distribution can
change over time yielding the phenomenon of concept drift
o The real concept drift refers to changes in the conditional distribution of the output
(i.e., target variable) given the input (input features)
• Concept drift types:
Time
Datamean
Sudden/abru
pt
Incremental Gradual
Reoccurring
concepts
Outlier
(not concept drift)
Disambiguation
such as
“medal olympics
2014”
Seasonal change
such as
“black Friday
2014”
Breaking news
such as
“idaho bus crash
investigation”
Disambiguation
such as
“cikm conference
2014”
Change Detection Techniques
24. Learn
reformulation
model M
User Behavior
Logs
ti
Incoming User
Behavior logs
Timeline
Detect changes in model M
If change detected
else Do Nothing
ti+w1 ti+w1+w2
Alarm:
Change of user
satisfaction
detected
for pairs :
{<Qi,
SERPi>}1<i<n
25. Learn
reformulation
model M
User Behavior
Logs
t0
Incoming User
Behavior Logs
Timeline
Detect changes in model M
If change detected
else Do Nothing
ti ti+ t
1) List of reformulation terms
per query
2) List of URLs per
reformulation
Alarm:
Change of user
satisfaction
detected
for pairs :
{<Qi,
SERPi>}1<i<n
26. o Dataset consists of 6 months
of the behavioral log data
from a commercial search
engine
o The training window size is
one month
o The test window size is two
weeks
Experimentation
29. o We successfully leveraged the concept drift detection
techniques to detect changes in user satisfaction
o The proposed technique works in unsupervised way
o Large scale evaluation has been performed
o Classification of the drift type is needed
o Prediction of the lifetime of the drift would help
Conclusion and Future Work
32. o We successfully leveraged the concept drift detection
techniques
o The proposed technique works in unsupervised way
o Large scale evaluation has been performed
o Classification of the drift type is needed
o Prediction of the lifetime of the drift would help
Conclusion and Future Work
Editor's Notes
In our work we use probability to reformulate a query as a sign of user satisfaction or dissatisfaction
Let revisit our example where the user wanted to visit CIKM