Slides from an invited lecture at the Netherlands Research School for Information and Knowledge Systems.

1. Yes
Appears
Useful?
Attractive?
Relevant?
Continue
on SERP?
Continue?
Examine Snippet
Click Document
Assess Document
Save Document
No
Yes
No
No
No
Yes Yes
Yes
No
Interactive Information Retrieval
A Quick Introduction: Modelling User Behaviours
SIKS Course: Advances in IR 5th October 2021
David Maxwell (@maxwelld90)

2. Who am I?
2
Glasgow
§ Postdoctoral Researcher
Delft University of Technology
§ PhD in Interactive IR
University of Glasgow (2019)
Modelling Search and Stopping in
Interactive Information Retrieval
David Martin Maxwell
School of Computing Science
College of Science and Engineering
University of Glasgow

3. Yes
Appears
Useful?
Attractive?
Relevant?
Continue
on SERP?
Continue?
Examine Snippet
Click Document
Assess Document
Save Document
No
Yes
No
No
No
Yes Yes
Yes
No
Interactive Information Retrieval
A Quick Introduction: Modelling User Behaviours
SIKS Course: Advances in IR 5th October 2021
David Maxwell (@maxwelld90)

4. Who do we build Search Engines for?
Information Seekers/Users
Individual(s) searching for information

5. Considering the User
§ Literature in IR has often focused on the system-side
§ Retrieval models, improving ranking, efficiency, etc.
§ But we develop search engines for users!
§ How do users behave?
§ How does an interface change behaviour?
§ How can we better support users?
§ We need to better understand complex
interactions between a user and system
5

6. What is Interactive IR?
6
“The area of interactive information retrieval covers
research related to studying and assisting these diverse
end users of information access and retrieval systems.”
Ian Ruthven
University of Strathclyde
“... the interactive approach to IR has led to a
focus on the user-oriented activities of query
formulation and reformulation, and inspection and
judgement of retrieved items ...”
Nick Belkin
Rutgers University
“In interactive information retrieval, users
are typically studied along with their
interactions with systems and information.”
Diane Kelly
University of Tennessee

7. Essential Reading (Too many to list here)
7
A probability ranking principle for interactive
information retrieval
Norbert Fuhr
Received: 14 September 2007 / Accepted: 15 January 2008 / Published online: 7 February 2008
! Springer Science+Business Media, LLC 2008
Abstract The classical Probability Ranking Principle (PRP) forms the theoretical basis
for probabilistic Information Retrieval (IR) models, which are dominating IR theory since
about 20 years. However, the assumptions underlying the PRP often do not hold, and its
view is too narrow for interactive information retrieval (IIR). In this article, a new theo-
retical framework for interactive retrieval is proposed: The basic idea is that during IIR, a
user moves between situations. In each situation, the system presents to the user a list of
choices, about which s/he has to decide, and the first positive decision moves the user to a
new situation. Each choice is associated with a number of cost and probability parameters.
Based on these parameters, an optimum ordering of the choices can the derived—the PRP
for IIR. The relationship of this rule to the classical PRP is described, and issues of further
research are pointed out.
Keywords Probabilistic retrieval ! Interactive retrieval ! Optimum retrieval rule
1 Introduction
Inf Retrieval (2008) 11:251–265
DOI 10.1007/s10791-008-9045-0
Information Foraging
Peter Pirolli
and
Stuart K. Card
UIR Technical Report
Funded in part by the Office of Naval Research
January 1999
Methods for Evaluating Interactive Information
Retrieval Systems for Users
Diane Kelly, 2009 (FTIR)
A Probability Ranking Principle for Interactive
Information Retrieval
Norbert Fuhr, 2008 (IRJ)
The Economics in Interactive Information Retrieval
Leif Azzopardi, 2011 (SIGIR)
Information Foraging
Peter Pirolli (pictured) and Stuart Card, 1999 (Psy. Review)

8. Lecture Outline
§ We’ll be considering IIR from a modelling perspective
8
Session 2 (14:30-15:15)
Session 1 (13:30-14:15)
Part I
System- to
User-Sided
Research
Part II
The Interactive
IR Process
Part III
Conceptual
Modelling of
Interactive IR
Part V
Evaluation and the Simulation of Interaction
Walkthrough: conducting a simulated analysis of
searcher behaviour
Part IV
Theoretical
Models of
Search

9. Just a Note
§ Interactive Information Retrieval is a huge area of active
research – with many different facets and areas
§ We’re only going to be looking at a small subset of research
§ We won’t be looking at user studies, for example
§ When we say “model” when I have the floor, we refer to a
model of a user’s interactions a searcher undertakes – not
some kind of retrieval model
9

10. System- to User-Sided Research
What is the difference?
PART I

11. System vs. User-Sided Search
11
System-Sided
Evaluation
Ranking,
efficiency,
etc.
Concerned with the
development of retrieval
models, efficiency
improvements, etc.

12. System vs. User-Sided Search
11
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.

13. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Document Corpus
Collection of documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

14. System vs. User-Sided Search
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Document Corpus
Collection of documents
Indexing Process
Converting to an index
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.
11

15. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Document Corpus
Collection of documents
Indexing Process
Converting to an index
Index
Various data structures
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

16. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Document Corpus
Collection of documents
Retrieval Engine
Returns a (ranked) list of
documents, given an index,
retrieval model and query
Indexing Process
Converting to an index
Index
Various data structures
Retrieval Model
Scores documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

17. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Document Corpus
Collection of documents
Retrieval Engine
Returns a (ranked) list of
documents, given an index,
retrieval model and query
Indexing Process
Converting to an index
Batch Queries
For system evaluation
Index
Various data structures
Retrieval Model
Scores documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

18. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Document Corpus
Collection of documents
Retrieval Engine
Returns a (ranked) list of
documents, given an index,
retrieval model and query
Indexing Process
Converting to an index
Judgements
Created by assessors
Batch Queries
For system evaluation
Index
Various data structures
Retrieval Model
Scores documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

19. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Interface/SERP
Generation of a Search
Engine Results Page (SERP)
to display matching results
Document Corpus
Collection of documents
Retrieval Engine
Returns a (ranked) list of
documents, given an index,
retrieval model and query
Indexing Process
Converting to an index
Judgements
Created by assessors
Batch Queries
For system evaluation
Index
Various data structures
Retrieval Model
Scores documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

20. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Interface/SERP
Generation of a Search
Engine Results Page (SERP)
to display matching results
Document Corpus
Collection of documents
Retrieval Engine
Returns a (ranked) list of
documents, given an index,
retrieval model and query
Indexing Process
Converting to an index
Judgements
Created by assessors
Searchers
With an information need,
seeking to satisfy said need
Batch Queries
For system evaluation
Index
Various data structures
Retrieval Model
Scores documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

21. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Interface/SERP
Generation of a Search
Engine Results Page (SERP)
to display matching results
Document Corpus
Collection of documents
Retrieval Engine
Returns a (ranked) list of
documents, given an index,
retrieval model and query
Indexing Process
Converting to an index
Information Need
What to search for
Judgements
Created by assessors
Searchers
With an information need,
seeking to satisfy said need
Batch Queries
For system evaluation
Index
Various data structures
Retrieval Model
Scores documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

22. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Interface/SERP
Generation of a Search
Engine Results Page (SERP)
to display matching results
Document Corpus
Collection of documents
Retrieval Engine
Returns a (ranked) list of
documents, given an index,
retrieval model and query
Indexing Process
Converting to an index
Information Need
What to search for
Query/Queries
Information need in term(s)
Judgements
Created by assessors
Searchers
With an information need,
seeking to satisfy said need
Batch Queries
For system evaluation
Index
Various data structures
Retrieval Model
Scores documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

23. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Interface/SERP
Generation of a Search
Engine Results Page (SERP)
to display matching results
Document Corpus
Collection of documents
Retrieval Engine
Returns a (ranked) list of
documents, given an index,
retrieval model and query
Indexing Process
Converting to an index
Information Need
What to search for
Query/Queries
Information need in term(s)
Judgements
Created by assessors
Searchers
With an information need,
seeking to satisfy said need
Interaction
Clicking links, examining...
Batch Queries
For system evaluation
Index
Various data structures
Retrieval Model
Scores documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.

24. System vs. User-Sided Search
11
Concerned with the
development of retrieval
models, efficiency
improvements, etc.
Concerned with the
examination of the
interactions between a
system and searcher, the
presentation of results, etc.
Interface/SERP
Generation of a Search
Engine Results Page (SERP)
to display matching results
Document Corpus
Collection of documents
Retrieval Engine
Returns a (ranked) list of
documents, given an index,
retrieval model and query
Indexing Process
Converting to an index
Information Need
What to search for
Query/Queries
Information need in term(s)
Judgements
Created by assessors
Searchers
With an information need,
seeking to satisfy said need
Interaction
Clicking links, examining...
Batch Queries
For system evaluation
Index
Various data structures
Retrieval Model
Scores documents
System-Sided
Evaluation
Ranking,
efficiency,
etc.
User-Sided
Evaluation
Interaction,
presentation,
etc.
Interaction Cycle
“Classical IR”
Research

25. The (Interactive) IR Spectrum
“Archetypical IIR Study”
System Focused User/Searcher Focused
1
2
3
4
5
6
7
8
Figure adapted from Diane Kelly’s IIR evaluation book. With permission. 12
§ Users/searchers are involved in different studies to
varying degrees (or not at all!) – this spectrum is a handy
way to categorise them

26. The (Interactive) IR Spectrum
“Archetypical IIR Study”
System Focused User/Searcher Focused
TREC-style studies
1
2
3
4
5
6
7
8
Figure adapted from Diane Kelly’s IIR evaluation book. With permission. 13
§ “TREC-style” studies were for system-sided research
§ Assessors create the relevance judgements, but no real
interactions are observed per se

27. The (Interactive) IR Spectrum
“Archetypical IIR Study”
System Focused User/Searcher Focused
TREC-style studies
“User” makes
relevance assessments
1
2
3
4
5
6
7
8
Figure adapted from Diane Kelly’s IIR evaluation book. With permission. 14
§ What if you use a collection (i.e., web-based experiments)
where no relevance judgements are available?
§ Typically used for the creation of document collections (i.e., is
this document relevant to the information need?)

28. The (Interactive) IR Spectrum
“Archetypical IIR Study”
System Focused User/Searcher Focused
TREC-style studies
“User” makes
relevance assessments
Filtering and SDI
1
2
3
4
5
6
7
8
Log analysis
Figure adapted from Diane Kelly’s IIR evaluation book. With permission. 15
§ The dissemination of “transaction logs” from search engines to
improve ranking models, etc.
§ Assumptions made on user intention – huge volumes of data
allow researches to identify important regularities

29. The (Interactive) IR Spectrum
“Archetypical IIR Study”
System Focused User/Searcher Focused
TREC-style studies
“User” makes
relevance assessments
Log analysis
Filtering and SDI
1
2
3
4
5
6
7
8
TREC interactive studies
Figure adapted from Diane Kelly’s IIR evaluation book. With permission. 16
§ The typical IIR study – some system/interface is
evaluated, and we observe the searcher’s behaviour
§ Can be behavioural (interactions) or experience (surveys, etc.)
§ Typically report both system and human-based measures

30. The (Interactive) IR Spectrum
“Archetypical IIR Study”
System Focused User/Searcher Focused
TREC-style studies
“User” makes
relevance assessments
Log analysis
Filtering and SDI TREC interactive studies
1
2
3
4
5
6
7
8
Experimental
information behaviour
Information seeking
behaviour in context
Information seeking
behaviour with IR systems
Figure adapted from Diane Kelly’s IIR evaluation book. With permission. 17
§ 6 Isolation of specific components (i.e., controlling what
results are returned) – typically used in psychology studies
§ 8 Human-centric studies, where qualitative surveys and
interviews are typically conducted

31. Classical IR: The Cranfield Experiments
§ Much of the classical IR research follows
the Cranfield experiments devised by
Cyril Cleverdon at Cranfield university
§ Concept of a corpus, a set of
information needs, and judgements
Did you know that Cranfield is the only university in the world with a functional airport? 18
“…a laboratory type situation where, freed as far as
possible from the contamination of operational variables,
the performance of index languages could be considered
in isolation.” Cleverdon (1991)
Image
credit:
https://www.cranfield.ac.uk/press/news-2016/cranfield-
university-announces-plans-for-festival-of-flight

32. Assumptions of Cranfield
§ Assumes a static information need
§ Once a searcher starts, their need does not evolve over time
§ Representative of an entire population
§ Searchers assume the same documents are relevant
§ The list of documents is total and complete
§ All relevant documents have been identified beforehand
19
While good for (simplifying) experimentation, are these assumptions realistic?

33. § Assumptions are good for reproducible, system-sided
research; lacking for user-sided research
§ Carried across to Information Retrieval evaluation fora
(e.g., TREC, NTCIR, CLEF…)
§ Over the years have provided numerous test collections and
topics for assisting in promoting reproducible research
k
Cranfield “User Models”
20
Researchers often state that Cranfield neglects the user; they
don’t! They actually abstract the user…
?

34. The “Cranfield Style” Searcher Model
Cutoff k
reached?
Issue Query
Consider Relevant
No
Click Document
Click Summary
No more
topics?
Yes
Yes
No
21

35. “Cranfield Style” Searcher Assumptions
22
Cutoff k
reached?
Issue Query
Consider Relevant
No
Click Document
Click Summary
No more
topics?
Yes
Yes
No

36. “Cranfield Style” Searcher Assumptions
23
Cutoff k
reached?
Issue Query
Consider Relevant
No
Click Document
Click Summary
No more
topics?
Yes
Yes
No
A single query is
issued for each topic

37. “Cranfield Style” Searcher Assumptions
24
Cutoff k
reached?
Issue Query
Consider Relevant
No
Click Document
Click Summary
No more
topics?
Yes
Yes
No
A single query is
issued for each topic
Every document is inspected,
regardless of relevance to the
topic being examined

38. “Cranfield Style” Searcher Assumptions
25
Cutoff k
reached?
Issue Query
Consider Relevant
No
Click Document
Click Summary
No more
topics?
Yes
Yes
No
A single query is
issued for each topic
Every document is inspected,
regardless of relevance to the
topic being examined
A user will “examine”
to a fixed depth of k

39. So what is More Realistic?
§ Interactive Information Retrieval saves the day…
§ We begin to shift away from these generally accepted
assumptions for Information Retrieval evaluation
§ We start to consider more realistic (but more complex)
interaction models to better explain what a user does
when interacting with a search system and/or information
26

40. The Interactive IR Process
Clicks, hovers, queries, and more
PART II

41. The Interactive IR Process
Retrieval Engine
28

42. The Interactive IR Process
Retrieval Engine
29

43. The Interactive IR Process
Retrieval Engine
Information Need
What to search for
29

44. The Interactive IR Process
Retrieval Engine
Information Need
What to search for
Query/Queries
Information need in term(s)
29

45. The Interactive IR Process
Interface/SERP
Retrieval Engine
Information Need
What to search for
Query/Queries
Information need in term(s)
Examination
29

46. The Interactive IR Process
Interface/SERP
Retrieval Engine
Information Need
What to search for
Query/Queries
Information need in term(s)
Click
Examination
Attractive?
29

47. The Interactive IR Process
Interface/SERP
Retrieval Engine
Information Need
What to search for
Query/Queries
Information need in term(s)
Document(s)
Click
The word
"Canberra" is
popularly
claimed to
derive from the word Kam-
bera or Canberry, which is
claimed to mean "meeting
place" in Ngunnawal, one of
the Indigenous languages
spoken in the district by...
Examination
Attractive?
29

48. The Interactive IR Process
Interface/SERP
Retrieval Engine
Information Need
What to search for
Query/Queries
Information need in term(s)
Document(s)
Click
The word
"Canberra" is
popularly
claimed to
derive from the word Kam-
bera or Canberry, which is
claimed to mean "meeting
place" in Ngunnawal, one of
the Indigenous languages
spoken in the district by...
Relevant?
Examination
Examination
Attractive?
29

49. The Interactive IR Process
Interface/SERP
Retrieval Engine
Information Need
What to search for
Query/Queries
Information need in term(s)
Document(s)
Click
STOP
The word
"Canberra" is
popularly
claimed to
derive from the word Kam-
bera or Canberry, which is
claimed to mean "meeting
place" in Ngunnawal, one of
the Indigenous languages
spoken in the district by...
Relevant?
Examination
Examination
Attractive?
29

50. The Interactive IR Process
Interface/SERP
Retrieval Engine
Information Need
What to search for
Query/Queries
Information need in term(s)
Document(s)
Examination
Attractive?
Click
STOP
STOP
The word
"Canberra" is
popularly
claimed to
derive from the word Kam-
bera or Canberry, which is
claimed to mean "meeting
place" in Ngunnawal, one of
the Indigenous languages
spoken in the district by...
Relevant?
Examination
29

51. The Interactive IR Process
Interface/SERP
Retrieval Engine
Information Need
What to search for
Query/Queries
Information need in term(s)
Document(s)
Reformulation
Examination
Attractive?
Click
STOP
STOP
The word
"Canberra" is
popularly
claimed to
derive from the word Kam-
bera or Canberry, which is
claimed to mean "meeting
place" in Ngunnawal, one of
the Indigenous languages
spoken in the district by...
Relevant?
Examination
29

52. The Information Need
§ The reason why a searcher approaches a retrieval system!
The Anomalous State of Knowledge (see Belkin)
A knowledge gap, or inconsistency with the world…
§ Is considered to be
dynamic (changes) as a
search session progresses
30

53. SERPs, Sessions, and Interactions
31
Canberra - Wikipedia
https://en.wikipedia.org/wiki/Canberra
Canberra is the capital city of Australia. With a population of 403,468, it is Aus-
tralia's largest inland city and the eighth-largest city overall. The city is located...
VisitCanberra: Canberra Holidays, Accommodation & Things...
https://visitcanberra.com.au/
Discover things to do in Canberra with our guide. Experience culture at the Na-
tional Portrait Gallery and the National Gallery of Australia, or visit the...
Canberra Airport | Arrivals, Departures, Lounges, Transport...
https://www.canberraairport.com.au/
Official website for Canberra Airport - The latest information on flights, parking,
transport and more. View live information on arrivals and departures.
canberra australia
Example Search Engine Results Page (SERP)
Query Terms
Title
Canberra
Capital of Australia
Canberra is the capital city of
Australia. With a population of
403,468, it is Australia’s larg-
est inland city and the
eighth-largest city overall.
Wikipedia
Left Rail (Result Summaries)
Right Rail
Source Snippet Fragments
Information Card
Result
Summary

54. SERPs, Sessions, and Interactions
32
Canberra - Wikipedia
https://en.wikipedia.org/wiki/Canberra
Canberra is the capital city of Australia. With a population of 403,468, it is Aus-
tralia's largest inland city and the eighth-largest city overall. The city is located...
VisitCanberra: Canberra Holidays, Accommodation & Things...
https://visitcanberra.com.au/
Discover things to do in Canberra with our guide. Experience culture at the Na-
tional Portrait Gallery and the National Gallery of Australia, or visit the...
Canberra Airport | Arrivals, Departures, Lounges, Transport...
https://www.canberraairport.com.au/
Official website for Canberra Airport - The latest information on flights, parking,
transport and more. View live information on arrivals and departures.
canberra australia
Example Search Engine Results Page (SERP)
Query Terms
Title
Canberra
Capital of Australia
Canberra is the capital city of
Australia. With a population of
403,468, it is Australia’s larg-
est inland city and the
eighth-largest city overall.
Wikipedia
Left Rail (Result Summaries)
Right Rail
Source Snippet Fragments
Information Card
Result
Summary
Interactions are recorded and
stored for post-hoc analysis

55. SERPs, Sessions, and Interactions
33
Canberra - Wikipedia
https://en.wikipedia.org/wiki/Canberra
Canberra is the capital city of Australia. With a population of 403,468, it is Aus-
tralia's largest inland city and the eighth-largest city overall. The city is located...
VisitCanberra: Canberra Holidays, Accommodation & Things...
https://visitcanberra.com.au/
Discover things to do in Canberra with our guide. Experience culture at the Na-
tional Portrait Gallery and the National Gallery of Australia, or visit the...
Canberra Airport | Arrivals, Departures, Lounges, Transport...
https://www.canberraairport.com.au/
Official website for Canberra Airport - The latest information on flights, parking,
transport and more. View live information on arrivals and departures.
canberra australia
Example Search Engine Results Page (SERP)
Query Terms
Title
Canberra
Capital of Australia
Canberra is the capital city of
Australia. With a population of
403,468, it is Australia’s larg-
est inland city and the
eighth-largest city overall.
Wikipedia
Left Rail (Result Summaries)
Right Rail
Source Snippet Fragments
Information Card
Result
Summary
Search Sessions often
constitute multiple queries

56. Search is Inherently Interactive
§ We know that the search process is not rigid!
§ Information needs are dynamic, and vary as a searcher
consumes information
§ Thinking about the complexity of a SERP and the interactions,
the basic searcher model is inadequate for demonstrating
what actually takes place when searching
§ Researchers have devised a number of expanded conceptual
and theoretical models to better explain IIR
34

57. Conceptual Modelling of Interactive IR
How can we better represent the search process?
PART III

58. Conceptual Models of Search
§ A conceptual model of search attempts to capture the
key interactions that take place during a search session
§ Being conceptual, they act as scaffolding – you can take
the scaffolding, and build all sorts of “user interaction models”
with them (instantiate each block in different ways)
Conceptual models differ from theoretical models; see later! 36
Write Slide Scream into Void Complete?

59. Expanded Conceptual Models
Adapted (with permission) from Baskaya et al. (see CIKM 2013 proceedings) 37
Issue Query Examine Snippet
Relevant?
Yes
Attractive?
Stop
Session?
Read Document
Continue
Examining
SERP?
No
Yes Yes
No
No
No
Yes

60. Expanded Conceptual Models
Adapted (with permission) from Baskaya et al. (see CIKM 2013 proceedings) 38
P=1 P<=1 P=1
P=1
P<=1
P<=1
P<=1
P<=1
P<=1
Formulate Query Scan a Snippet Click a Link Read a Document
Judge Document
Relevance
Stop Session
P<=1

61. Expanded Conceptual Models
§ General flow of the
searcher is the same as
before
§ Allowed searcher to
select which summary
to read – non-linear!
§ Also incorporates
ability to select a
search system to use
For more information, have a look at Thomas et al. (IIiX, precursor to CHIIR, in 2014) 39
Enticed by
summary i?
Select System Enter Query Choose position i
Evaluate summary i
Click
summary
link?
Read (part of)
document
End query?
End session?
Decide
next action
si
ri
No
Yes
Yes
No
Yes
No
Yes
No
Change
query
Change retrieval system

62. Expanded Conceptual Models
§ The Complex Searcher
Model – adapted from
observing logs and previous
conceptual models
§ More on this later
§ We can consider blocks in
isolation, or as part of the
entire process
https://www.dmax.org.uk/thesis/ 40
Yes
Select
Query
Out of queries
Appears
Useful?
Attractive?
Relevant?
Continue
on SERP?
Continue?
Examine Topic Generate Queries Issue Query
View SERP
Examine Snippet
Click Document
Assess Document
Mark Document
No
Yes
No
No
No
Yes Yes
Yes
No

63. Theoretical Models of Search
Providing us with predictive power
PART IV

64. Theoretical Models of Search
§ IIR researchers have proposed mathematically grounded
models that provide us with a descriptive, predictive ability
to explain how and why searchers behave in a given way
§ Such models have limitations, too!
§ Assumptions in human behaviour (behaving rationally)
§ Mathematically-based can be considered closed-form and can
make it hard to model the complex phenomena1
1Fishwick (1995) outlines simulation as a means for permitting complex phenomena; see later. 43

65. Theoretical Models of Search
44
§ Three competing theoretical models have been proposed…
All three theories have been shown to be mathematically equiv. See Azzopardi and Zuccon (2015 ICTIR)
Interactive
Probability
Ranking
Principle
Norbert Fuhr, 2008
Search
Economic
Theory
Leif Azzopardi, 2011
Information
Foraging
Theory
Peter Pirolli and
Stuart Card, 1999
Expanding the PRP Economic theory Animal behaviour

66. The Berrypicking Model
§ A well known model where searchers are considered
analogous to foragers, scavenging for food in the wild
See Bates (1989) 45

67. The Berrypicking Model
§ A well known model where searchers are considered
analogous to foragers, scavenging for food in the wild
See Bates (1989). 45

68. The Berrypicking Model
§ A well known model where searchers are considered
analogous to foragers, scavenging for food in the wild
See Bates (1989). 45

69. The Berrypicking Model
§ A well known model where searchers are considered
analogous to foragers, scavenging for food in the wild
See Bates (1989). 45

70. The Berrypicking Model
§ A well known model where searchers are considered
analogous to foragers, scavenging for food in the wild
See Bates (1989). 45

71. The Berrypicking Model
§ A well known model where searchers are considered
analogous to foragers, scavenging for food in the wild
See Bates (1989). 45

72. The Berrypicking Model
§ A well known model where searchers are considered
analogous to foragers, scavenging for food in the wild
See Bates (1989). 45

73. The Berrypicking Model
§ A well known model where searchers are considered
analogous to foragers, scavenging for food in the wild
Bates does publish a later paper that discusses cost/benefit analyses, however. 46
§ Highly descriptive, but importantly, not predictive
§ You go for the juiciest berries, but the model does not provide a
rationale as to why (accruing gain)
§ How long should a forager spend in a given berry bush?
§ We need models that offer predictive power to answer this

74. Information Foraging Theory
§ Devised from Foraging Theory, the study
of how animals forage for food
§ Examining their behaviours, where they
attempt to maximise their gain (intake)
per unit of time (in order to survive)
47
A totally fascinating book; see Stephens and Krebs (1986)

75. Information Foraging Theory
§ Pirolli & Card applied Foraging Theory to search!
§ Foraging Theory costs of three models…
48
Diet Model Patch Model Scent Model

76. Forager
Patches and Scent
§ An area in which gains can be made is called a patch
§ A forager will follow a given scent to the patch, and make
decisions as to whether to head towards it, or once inside,
when to leave it
49

77. Forager Patch
Patches and Scent
§ An area in which gains can be made is called a patch
§ A forager will follow a given scent to the patch, and make
decisions as to whether to head towards it, or once inside,
when to leave it
49

78. Forager
Scent (Pollen)
Patch
Patches and Scent
§ An area in which gains can be made is called a patch
§ A forager will follow a given scent to the patch, and make
decisions as to whether to head towards it, or once inside,
when to leave it
49

79. Forager
Scent (Pollen)
Patch
Patches and Scent
§ An area in which gains can be made is called a patch
§ A forager will follow a given scent to the patch, and make
decisions as to whether to head towards it, or once inside,
when to leave it
49

80. Forager
Scent (Pollen)
Patch
Beetween patch time
Patches and Scent
§ An area in which gains can be made is called a patch
§ A forager will follow a given scent to the patch, and make
decisions as to whether to head towards it, or once inside,
when to leave it
49

81. Forager
Scent (Pollen)
Patch
Beetween patch time
Within patch time
Patches and Scent
§ An area in which gains can be made is called a patch
§ A forager will follow a given scent to the patch, and make
decisions as to whether to head towards it, or once inside,
when to leave it
49

82. Forager
Scent (Pollen)
Patch
Beetween patch time
Within patch time STOP
Patches and Scent
§ An area in which gains can be made is called a patch
§ A forager will follow a given scent to the patch, and make
decisions as to whether to head towards it, or once inside,
when to leave it
49

83. Forager
Scent (Pollen)
Patch
Beetween patch time
Within patch time STOP
Patches and Scent
§ An area in which gains can be made is called a patch
§ A forager will follow a given scent to the patch, and make
decisions as to whether to head towards it, or once inside,
when to leave it
49

84. Foragers
Patches, Scent, and Search
§ How does search fit into this beelievable theory?
50

85. Foragers
Patches
Patches, Scent, and Search
§ How does search fit into this beelievable theory?
50

86. Interface/SERP
Foragers
Patches
Patches, Scent, and Search
§ How does search fit into this beelievable theory?
50

87. Interface/SERP
Foragers
Patches
Within Patch Time
Patches, Scent, and Search
§ How does search fit into this beelievable theory?
50

88. Interface/SERP
Foragers
Patches
search query
Between Patch Time
Within Patch Time
Patches, Scent, and Search
§ How does search fit into this beelievable theory?
50

89. Interface/SERP
Foragers
Patches
search query
Between Patch Time
Within Patch Time
Scent?
Patches, Scent, and Search
§ How does search fit into this beelievable theory?
50

90. Predictive Power of IFT
§ We can use IFT to predict when
someone should stop examining
results on a SERP (for example)
§ We can use the Marginal Value
Theorem to predict when you should
stop and leave a patch
§ This is called the optimal stopping
point – gain diminishes after this!
51
Cumulative
Gain
(CG)
Time

91. Predictive Power of IFT
§ We can use IFT to predict when
someone should stop examining
results on a SERP (for example)
§ We can use the Marginal Value
Theorem to predict when you should
stop and leave a patch
§ This is called the optimal stopping
point – gain diminishes after this!
51
Cumulative
Gain
(CG)
Time

92. Predictive Power of IFT
§ We can use IFT to predict when
someone should stop examining
results on a SERP (for example)
§ We can use the Marginal Value
Theorem to predict when you should
stop and leave a patch
§ This is called the optimal stopping
point – gain diminishes after this!
51
Cumulative
Gain
(CG)
Time
Gain Curve

93. Predictive Power of IFT
§ We can use IFT to predict when
someone should stop examining
results on a SERP (for example)
§ We can use the Marginal Value
Theorem to predict when you should
stop and leave a patch
§ This is called the optimal stopping
point – gain diminishes after this!
51
Cumulative
Gain
(CG)
Time
Gain Curve
Between Patch

94. Predictive Power of IFT
§ We can use IFT to predict when
someone should stop examining
results on a SERP (for example)
§ We can use the Marginal Value
Theorem to predict when you should
stop and leave a patch
§ This is called the optimal stopping
point – gain diminishes after this!
51
Cumulative
Gain
(CG)
Time
Within Patch
Gain Curve
Between Patch

95. Predictive Power of IFT
§ We can use IFT to predict when
someone should stop examining
results on a SERP (for example)
§ We can use the Marginal Value
Theorem to predict when you should
stop and leave a patch
§ This is called the optimal stopping
point – gain diminishes after this!
51
Cumulative
Gain
(CG)
Time
Within Patch
Gain Curve
Between Patch

96. Predictive Power of IFT
§ We can use IFT to predict when
someone should stop examining
results on a SERP (for example)
§ We can use the Marginal Value
Theorem to predict when you should
stop and leave a patch
§ This is called the optimal stopping
point – gain diminishes after this!
51
Cumulative
Gain
(CG)
Time
Within Patch
A
v
e
r
a
g
e
R
a
t
e
o
f
G
a
i
n
Gain Curve
Between Patch

97. Predictive Power of IFT
§ We can use IFT to predict when
someone should stop examining
results on a SERP (for example)
§ We can use the Marginal Value
Theorem to predict when you should
stop and leave a patch
§ This is called the optimal stopping
point – gain diminishes after this!
51
Cumulative
Gain
(CG)
Time
Within Patch
A
v
e
r
a
g
e
R
a
t
e
o
f
G
a
i
n
Gain Curve
Between Patch

98. Predictive Power of IFT
§ We can use IFT to predict when
someone should stop examining
results on a SERP (for example)
§ We can use the Marginal Value
Theorem to predict when you should
stop and leave a patch
§ This is called the optimal stopping
point – gain diminishes after this!
51
Cumulative
Gain
(CG)
Time
Within Patch
STOP
A
v
e
r
a
g
e
R
a
t
e
o
f
G
a
i
n
Gain Curve
Between Patch
Prediction: stop
at this point!

99. Predicting Other Behaviours
§ IFT can predict a variety of other behaviours too – it’s
how you apply it that is important
§ Whether to enter a patch/SERP, etc…
§ Competing theories (e.g., SET) have also been used to
predict various search behaviours
§ For example, query length vs. gain trade-offs – what is the optimal
query length for a searcher to issue?1
§ Deals with cost/benefit trade-offs – what is most efficient?
1See the tutorial by Azzopardi and Zuccon on developing economic models. 52

100. Evaluation and Simulation
How can we evaluate our models of search?
PART V

101. Why is this Important?
§ Theoretical models provide us with an underpinning and
explanation for (rational) searcher behaviours
§ Conceptual models are based on what theoretical models
suggest plus real-world observations of searcher behaviours
to formalise the steps and decisions taken
§ Together, we have a strong set of tools to provide a credible
explanation of the IIR process – but how do we know they
are any good?
54

102. How do we Evaluate these Models?
§ Evaluation is important – how do we know they are
credible? How do we know they are useful?
§ We can evaluate these models through a combination of
user studies and the simulation of interaction
§ Following a long line of IR research using simulation
§ Offers the freedom to explore a wide range of scenarios (i.e., what
if experiments) all at a low cost, without searcher fatigue, etc.
Refer to Fishwisk (1995) for a detailed and nuanced argument for simulation. 55

103. The Simulation of Interaction
56
§ We can instantiate each of
the building blocks and
decision points in different
ways to see what happens
§ Studies have examined simulated
queries, browsing behaviours, cost
vs. time, session performance…
§ These experiments must
be properly grounded –
perhaps using interaction
data from a real-world study
Yes
Select
Query
Out of queries
Appears
Useful?
Attractive?
Relevant?
Continue
on SERP?
Continue?
Examine Topic Generate Queries Issue Query
View SERP
Examine Snippet
Click Document
Assess Document
Mark Document
No
Yes
No
No
No
Yes Yes
Yes
No