Slides of our SIGIR 2018 paper "Predicting User Knowledge Gain in Informational Search Sessions", which is presented in Ann Arbor, MI, US on July 9th, 2018.

1. Predicting User Knowledge Gain
in Informational Search Sessions
Ran Yu1
, Ujwal Gadiraju1
, Peter Holtz2
, Markus Rokicki1
, Philipp Kemkes1
, Stefan Dietze1,3
1. L3S Research Center, Leibniz Universität Hannover; Hannover, Germany
2. Leibniz-Institut für Wissensmedien (IWM); Tübingen, Germany
3. GESIS - Leibniz-Institute for the Social Sciences; Köln, Germany
@ranyu_zh yu@l3s.de

2. Search Engine for Serving Learning Needs
● Web search is frequently used to acquire new
knowledge & satisfy learning-related
objectives
● How does the knowledge of a user evolve
through the course of informational search
sessions and can we predict user knowledge
(change)?
Web Search Queries : Navigational, transactional or informational intents
[Broder, 2002]
.. in informational web search sessions, the intent of a user is to acquire
some information assumed to be present on one or more web pages ..
Intentional learning - “learning that is motivated by intentions and is goal
directed”, “cognitive process that have learning as a goal rather than an
accidental outcome”
2

3. Goal
Build models to detect user knowledge & knowledge gain:
● classifying the knowledge state (KS) at the end of the informational session with
respect to a particular information need into: low, moderate, high
● classifying the knowledge gain (KG) during the informational session into: low,
moderate, high
Knowledge on
SAL at 9:00?
Knowledge on
SAL at 9:50?
How did learning
happen?
3

4. e.g.
1. Eickhoff et al. investigated the correlation between several query and search
mission-related metrics with learning progress [2014]
2. Wu et al. predicted the difficulty of search tasks from query and mission-related
features [2012]
3. Collins-Thompson et al. investigated the aspects of search interaction which are
effective for supporting superior learning outcomes in vocabulary learning
scenario [2016, 2017,2018]
4. Zhang et al. explored using search behavior as an indicator for the domain
knowledge of a user [2015]
Prior works
4

5. Prior works
Summary:
● The learning related notions: knowledge gain, knowledge state, user engagement,
learning stage (based on e.g. Anderson and Krathwohl's taxonomy), expertise of a
topic, task difficulty
● Features extracted from: search session, search engine result page (SERP),
document, user behavior, eye-tracking data
● limited set of features
● addressing specific learning scenarios
● the generalizability of knowledge gain measures have not been
investigated
● no automated method for predicting user knowledge/gain.
5

6. Data Collection
1. Recruit workers from
crowdsourcing
platform
2. Pre-test to assess
worker’s initial
knowledge state on
the given topic
3. Direct the worker to
the SearchWell
platform to search and
browse documents
4. Post-test to assess
worker’s post
knowledge
6
Dataset available at: https://sites.google.com/view/predicting-user-knowledge

7. Topics & Information Needs
TREC 2014 Web
Track
7


8. Data Analysis
Knowledge gain is measured as the difference
between pre- and post-test score.
● 70% of users exhibited a knowledge gain
(KG)
● Negative relationship between KG of users
and topic popularity (avg. accuracy of
workers in knowledge tests) (R= -.87)
● Amount of time users actively spent on
web pages describes 7% of the variance
in their KG
● Query complexity explains 25% of the
variance in the KG of users
More results in:
Ujwal Gadiraju, Ran Yu, Stefan Dietze, Peter Holtz. Analyzing Knowledge Gain of
Users in Informational Search Sessions on the Web. ACM CHIIR 2018.
8

9. Generating class label
Generating classes: Group user knowledge state
(KS) and knowledge gain (KG) into {low, moderate,
high} group using Standard Deviation
Classification approach.
mean ± 0.5 SD
9

10. Considered Features
● Session, e.g. session duration, duration per query, ...
● Query, e.g. query length, query number, max query complexity, …
● SERP (Search engine result page), e.g. time on SERP, number of clicks, ...
● Browsing, e.g. number of pages, avg time per page, …
● Mouse, e.g. max scroll distance, number of mouseovers, ...
10

11. Models & configurations
Classifiers - Naive Bayes, Logistic regression, SVM, random forest, multilayer
perceptron. (Applied grid search to find the best parameters.)
Feature analysis & selection
● Pearson correlation between feature and KG (KS)≥ β (γ)
● Correlation between features < τ
Correlation
between
features
Correlation between feature & KG
11

12. Result - KG prediction
Overall: meaningful evidence for predicting
knowledge gain
Accuracy:
Efficiency:
245 distinct configurations * 10-fold cross
validation * 10 repetitions
Metrics:
● Accuracy (Accu) across all classes
● Precision (P), Recall (R), F1 (F1) score of
each class
Macro average of precision (P), recall (R),
and F1 (F1)
● Runtime in milliseconds.
RF
SVM NB
NB
12

13. Feature importance - KG
Individual features
● Browsing time related
● Page title length
● Amount of mouse movement
● Rank of the clicked documents
● ...
13
Feature categories
Browsing
Mouse SERP

14. Result - KS prediction
140 distinct configurations * 10-fold cross
validation * 10 repetitions
Metrics:
● Accuracy (Accu) across all classes
● Precision (P), Recall (R), F1 (F1) score of
each class
Macro average of precision (P), recall (R),
and F1 (F1)
● Runtime in milliseconds.
Overall: meaningful evidence for predicting
knowledge state. Outperforms baseline.
Accuracy:
Efficiency:
KSZhang
14
RF
MP SVM

15. Feature importance - KS
15
Individual features
● Query complexity
● Page title length
● Page title unique terms
● Query length
● Browsed page size
● ...
Feature categories
Query
Browsing

16. Discussion & Conclusions
Conclusions
● knowledge gain (state) can be predicted during informational search sessions
with a certain level of accuracy,
● performance of the knowledge gain prediction appears to be generally better,
suggesting that the
task is easier given the nature of our data, and
● the performance of the prediction approach is better for more extreme classes
Limitations
● limited duration of the search sessions reduce the predictive power of certain
features
● topic descriptions provided central keywords for the first query, which makes
the query features less distinguishable
16

17. Future works
● Conducting experiments to get data for more varied search sessions
● reproduce and refine the findings in more varied search sessions (e.g.
longer duration, procedural knowledge)
● investigate features of (multimedia) resources that user interacted
with
● Use predicted KS & KG for optimizing retrieval algorithms
17

18. Thank you!
Q&A
18

19. A Few More Details . . .
TEST QUESTION SELECTION:
● Using 100 workers per topic, and a larger pool of items
(~30); filter items that were too easy (>80% of the workers
got it right) or too hard/ambiguous (<20% got it right).
SAL-SESSIONS:
● Participation of only Level-3 CrowdFlower workers from
primarily English-speaking countries
● 50 workers per topic, filtered out workers who entered no
queries, workers who selected the same option
‘TRUE/FALSE’ for all items, those who did not complete the
post-session test ⇒ 420 workers
19