Relevance.ppt

© Tefko Saracevic 1
Relevance
in information science

Relevance
Preface
1975 & 1976
 Relevance: A Review of the Literature
and a Framework for Thinking on the
Notion in Information Science
2006
 Relevance: A Review of the Literature
and a Framework for Thinking on the
Notion in Information Science. Part II

Relevance
Purpose
trace the evolution of thinking on
relevance in information science for
the past three decades
provide an updated framework within
which the still widely dissonant ideas
on relevance might be interpreted
and related to one another

Relevance
Relevance & information
technology
Relevance around forever
Computers ~ 50 years
 internet ~ 30; web ~ 15
IT changed information activities
Systems designed to provide
relevant information
Invisible hand of relevance always
present

Relevance
… in information science
Information retrieval (IR) systems
offer their version of what may be
relevant
People go their way & asses relevance
The two worlds interact
Concern here:
 human world of relevance
 how IR deals with relevance NOT covered

Relevance
Significance of relevance
research
Contribution of relevance
scholarship:
 knowledge for knowledge sake
 better understanding of the notion of
relevance significant in itself
 potential for making information systems
better – more human oriented
 technological advances often dependent on
better understanding of underlying notions,
phenomena

Relevance
Historical note
Relevance came into IR unannounced
From very start after WWII IR was
about retrieval of relevant information
First concerns: about “false drops” –
non relevant retrievals
First direct recognition in 1955 with
proposal for precision & recall as
evaluation measures based on relevance

Relevance
Organization of presentation
1. What is the NATURE of relevance?
 1.1 Meaning?
 1.2 Theories?
 1.3 Models?
2. What are MANIFESTATIONS of
relevance?
3. What is the BEHAVIOR of
relevance?
4. What are the EFFECTS of
relevance?

Relevance
1.1 Meaning of relevance
Intuitively well understood
 same perception globally – “y’know”
 a “to” and context always present
Relevance:
 a relation between objects P & Q along
property R
 may also include a measure S of the strength
of connection

Relevance
… in information science
Relation between inf. or inf. objects
(Ps) & contexts (Qs) based on some
property R & measure of intensity S
But: relevance is not given – it is
established
Big questions & challenges
 How does relevance happen?
 Who does it, under what circumstances,
and how?

Relevance
Inference:
created - derived
Systems & automatons create
relevance
Users derive relevance
 but could be more complex & a continuum
 derive could involve more than topicality
 greater expertise = greater derivative powers
 create could involve “intelligence”
 also involves
 intent
 selection & interaction processes

Relevance
Summary:
Meaning of relevance
Relevance attributes involve:
 relation
 intention
 context
 internal; external
 inference
 selection
 interaction
 measurement

Relevance
1.2 Theories of relevance
Theories suggested in several fields
 logic – in deduction of inferences, to reject
fallacies – relevance logics
 philosophy – in phenomenology structure &
functioning of the “life-world” (Alfred Schutz)
 it is stratified & relevance is the principle for
stratification
 no single relevance but an interdependent
system of relevances (plural)
 thematic (topical), interpretational, &
motivational relevance

Relevance
… theories in communication
Sperber & Wilson: Relevance Theory
 based on inferential model of communication
 what must be relevant and why to an
individual with a single cognitive intention?
 posited a cognitive & a communicative
principle of relevance
 assessed in terms of cognitive effects &
processing effort
 individuals pick up the most relevant stimuli &
process them to maximize their relevance

Relevance
Summary:
Theories of relevance
IS did not develop indigenous theory
but a few “theories-on-loan” attempts
 Logic theories not applied, yet
 Schutz’s life-world theory used to some
extend in specifying manifestations &
models
 Sperber & Wilson’s theory used few
times as explanation & guide
 not tested

Relevance
1.3 Models of relevance
Reviews – also produced models
Syracuse school of relevance
 dynamic & situational model (Schamber, Eisenberg
& Nilan, 1990)
 connection with human information behavior
“Whole history of relevance” (Mizzaro, 1997)
 duality in modeling & studying
 documents & queries, 1959-1976
 dynamics & multidimensionality, 1977-present

Relevance
Split between system & user
models
Opposing views of IR: systems & users
 IR traditional model does not deal with users
Battle royal started by Dervin & Nilan
(1986)
 criticism of system viewpoint
 call for user orientation
Several user & interaction models
proposed – to reconcile, bridge
 still relevance has two basic models &
cultures & they map like Australia

Relevance
Stratified model
Situational
tasks; context...
A
d
a
p
t
a
t
i
o
n
Engineering
hardware; connections...
A
d
a
p
t
a
t
i
o
n
I
N
T
E
R
A
C
T
I
O
N
&
R
E
L
E
V
A
N
C
E
S
T
R
A
T
A
Surface level
U
s
e
o
f
i
n
f
o
r
m
a
t
i
o
n
Query
characteristics …
C
O
M
P
U
T
E
R
Affective
intent; motivation ...
Cognitive
knowledge; structure...
Processing
software; algorithms …
Content
inf. objects; representations...
U
S
E
R
INTERFACE
Situational
tasks; context...
A
d
a
p
t
a
t
i
o
n
Engineering
hardware; connections...
A
d
a
p
t
a
t
i
o
n
I
N
T
E
R
A
C
T
I
O
N
&
R
E
L
E
V
A
N
C
E
S
T
R
A
T
A
Surface level
U
s
e
o
f
i
n
f
o
r
m
a
t
i
o
n
Query
characteristics …
C
O
M
P
U
T
E
R
Affective
intent; motivation ...
Cognitive
knowledge; structure...
Processing
software; algorithms …
Content
inf. objects; representations...
U
S
E
R
INTERFACE

Relevance
User vs system models
“Informing systems design”
 became mantra of all relevance studies
“Tell us what to do and we will do it.”
 response from systems side
But “telling” is not that simple
Issue is not a conflict but:
 how can we make user & system side work
together for benefit of both?

Relevance
Summary:
relevance models
All IR & inf. seeking models have
relevance at their base
Traditional IR model has most
simplified –“weak”- version of relevance
 but with the weak model IR is successful
Variety of integrative models have
been proposed
 more complex models = increased challenge
to incorporate in practice

Relevance

Relevance
2. Manifestations of
relevance
“How many relevances in IR?” (Mizzaro, 1998)
 Several manifestations recognized since 1950s
 Issue: What given objects (Ps & Qs) are related
by what given property (Rs) as relation?
 [adjective] relevance or different name
Duality strikes again
 subject (topic, system) relevance
vs user (psychological, cognitive) relevance
 objective vs subjective relevance

Relevance
Issue of primacy:
weak and strong relevance
Does topical relevance underlie all
others?
 predictably two answers: yes and no
 in a strict correspondence between
query & answer topical is basic – weak
relevance
 if derivation is involved topical may not
be basic – strong relevance
 weak relevance is more associated with
systems
 strong more with people

Relevance
Beyond duality
Numerous other kinds of relevance
were identified
 for user relevances:
 psychological, cognitive, affective, situational,
socio-cognitive, pertinence, utility …
 for topical relevances:
 logical, systems, algorithmic, documentary,
bibliographic …
 each indicates different relations

Relevance
Summary:
Manifestations of relevance
There are a limited number of
manifestations – could be grouped:
 System or algorithmic relevance
 Topical or subject relevance
 Cognitive relevance or pertinence
 Situational relevance or utility
 Affective relevance
These are interdependent – they
feed on each other interactively

Relevance
3. Behavior of relevance
Relevance does not behave, people do
 how humans determine relevance of
information or information objects?
 reviewed only studies that have data
 some 30 studies – started in 1991
 related to information seeking & use studies &
implicit relevance studies – not reviewed
Pattern for review:
 [author] used [subjects] to do [tasks] in order to
study [object of research]

Relevance
Relevance clues
What makes information or information
objects relevant? What do people look
for in order to infer relevance?
 two approaches: topic & clues analysis
Clues research:
 uncover & classify attributes or criteria that
users concentrate on while making relevance
inferences
 usually on documents but also other objects

Relevance
Examples
Schamber (1991) interviewed 30 users of weather
information using different sources, from oral
reports to documents and maps in order to derive
and categorize their relevance criteria. She
identified 22 categories in 10 groups.
Barry (1994) interviewed 18 academic users (not
specified as being students or faculty) who had
requested an information search for documents
related to their work in order to categorize their
relevance criteria. She identified 23 categories in
7 groups

Relevance
Relevance dynamics
Do relevance inferences and criteria
change over time for the same user
and task, and if so, how?
As a user progresses through various
stages of a task
 the user’s cognitive state changes
 the task changes as well
 thus, something about relevance also is
changing.

Relevance
Relevance feedback
What factors affect the process of
relevance feedback?
 What types of feedback? How much is
feedback used?
Dealing here with manual not
automatic feedback
 behavior of people when involved in
manual feedback

Relevance
Summary on behavior
Caveats abound – nothing standardized
 still refreshing to see data
Some generalizations on clues:
 criteria finite in number, similar, but
different weights assigned
 Different users, tasks, progress in tasks, classes
of users = similar criteria = different weights
 Different ratings of relevance = similar criteria
= different weights.

Relevance
…summary clues
Clues criteria:
 content
 object
 validity
 use or situational match
 cognitive match
 affective match
 belief match

Relevance
… summary clues
Criteria are not independent; people
apply multiple criteria; they interact
 content (topic) criteria very important
but not sole – interact with others
 for search outputs value of results as a
whole critical
Visual information = faster inference
than textual information

Relevance
… summary dynamics
Inferences dependent on task stage
 criteria stable, selection changes
 Different stages = differing selections
but different stages = similar criteria =
different weights
 Increased focus = increased
discrimination = more stringent
relevance inferences
 What is topical changes with progress in
time and task

Relevance
… summary feedback
Several kinds
 search term, content, magnitude, tactics
Use of relevance feedback = increase
in performance
 however, used rarely in practice
Searching behavior different when
using feedback

Relevance
5. Effects of relevance
Works both ways: relevance affected by
and affects host of factors
Relevance judges
 What factors inherent in relevance judges make
a difference in relevance inferences?
 How large are and what affects individual
differences in relevance inferences?
 similar question asked for a number of information
activities – indexing, searching …
 most often studied: domain knowledge

Relevance
Relevance judgments
What factors affect relevance
judgments?
 short answer: a lot of them
 approach: classify into tables e.g.
 Schamber (1994) 80 factors, 6 categories
 Harter (1996) 24 factors, 4 categories
 different approach here:
 classify studies along basic assumptions in IR
evaluations

Relevance
Central assumptions
Relevance is:
 topical
 binary
 independent
 stable
 consistent
 if pooling: complete
Not to prove or disprove these assumptions
but to organize studies along questions

Relevance
Beyond topicality
Do people infer relevance based on
topicality only?
Other factors enter & interact
Only a few studies directly addressed
 Wang & Soergel (1998) 11 criteria for
selection, with topicality being top
 Xu & Chen (2006) in web searching: topicality &
novelty most significant, then reliability &
understandability

Relevance
Beyond binary
Are relevance inferences binary i.e.
relevant – not relevant? If not, what
gradation do people use in inferences
about relevance of information or
information objects?
Number of studies addressed this
 studied distributions of relevance
inferences
 regions of relevance

Relevance
Beyond independence
Are information objects assessed
independently of each other? Does the
order or size of the presentation affect
relevance judgments?
 Only a few studies on the questions
 Includes presentation of different
representations

Relevance
Beyond stability
Are relevance judgments stable as
tasks and other aspects change? Do
relevance inferences and criteria
change over time for the same user and
task, and if so how?
 mentioned already under dynamics
 judgments not completely stable, criteria are
 Plato: “Everything is flux.”

Relevance
Beyond consistency
Are relevance judgments consistent
among judges or group of judges?
 human judgments about anything informational
are not consistent, relevance included
Gull (1956) opened the Pandora box
 classic example of law of unintended
consequences
Some 6 studies addressed consistency
 subjects: experts, students

Relevance
But does it matter?
How does inconsistency in human
relevance judgments affect results of
IR evaluation?
 main contention by critics
Five studies 1968 – 2000 addressed
the question
 four also showed magnitude of agreement

Relevance
Summary: judges
Subject expertise accounts strongly
 higher expertise = higher agreement, less
differences
 lesser expertise = more leniency in judgment
 large variability in relevance inferences
by individuals
 same range as in other cognitive processes

Relevance
Summary: judgments
Relevance is measurable!
None of the 5 postulates hold
 but by simplifying relevance for labs IR
made significant advances
Relevance judgments are not binary
but are bimodal
 regions of low, middle and high relevance
 high peaks at both end
Order affect relevance judgment

Relevance
… summary: judgments
Consistency:
 Higher expertise = higher consistency =
more stringent. Lower expertise = lower
consistency= more encompassing
 overlap using different populations hovers
around 30%
 higher expertise up to 80%
 when 3rd, 4th … judge added overlap falls
 Higher expertise =larger overlap. Lower
expertise =smaller overlap. More judges =
less overlap.

Relevance
Summary: does it matter?
In lab conditions disagreement among
judges does not affect evaluation
 rank order of different IR systems changes
minimally
 Different judges = same relative performance (on
the average)
 swaps in ranking do occur = low probability
 but performance for individual topics differs
significantly
 law of averages kicks in

Relevance
Summary: measures
Users can use a variety of scales
 there is no “best” scale
 magnitude scales very appropriate but
hard to explain & analyze

Relevance
Epilogue
Many things changed in IR &
information science but goals the same
As to nature of relevance:
 marked progress in understanding
 little in theory
 diversification in models in models
As to manifestations:
 consensus there are several kinds of
relevance, grouped in a half dozen or so well
distinguished classes - interdependent

Relevance
… epilogue
As to behavior & effects:
 seen a number of experimental &
observational studies
 lifted the discourse beyond debate,
anecdotes to data interpretation
 but generalizations difficult – findings
should be treated as hypotheses

Relevance
… epilogue: reflections
Relevance is poor
 no funding for relevance research
 of studies with data less than 17% mentioned outside
funding, half from outside the US
 scholarship progressed sporadically & all over the
place
Globalization of IR – globalization of
relevance
 as relevance went global & to the masses many
& different research questions emerged

Relevance
… epilogue: reflections…
Proprietary IR – proprietary relevance
 major search engines proprietary =
relevance proprietary
 for innovation must study users & use, but
findings kept private
 relevance research into public & private branch
 paradox of the internet

Relevance
… epilogue: research agenda
- beyonds
Beyond behaviorism & black box
 many studies stimulus-response & no
diagnostics
 need to go use/adapt other approaches

Relevance
- beyonds
Beyond mantra
 “implication for system design”
 incorporating user concerns &
characteristics not that simple
 integration between user/cognitive &
systems approaches needed
 relevance research and IR research should
at least get engaged, if not married
 interactive research on the right track

Relevance
- beyonds
Beyond students
 ~70% of behavior & effect studies used
students as population
 not surprising – they are affordable
 we know a lot about student relevance
 does it generalize to other populations?

Relevance
In conclusion
Information technology & systems
will change dramatically
 even in the short run
 and in unforeseeable directions
But relevance is here to stay!

Relevance

Relevance
Acknowledgments
Ying Zhang & Yuelin Li for searches &
organization
Sandra Lanman for editing
Students in Spring 2006 class Human
Information Behavior for comments
 Jeanette de Richemond for corrections
SCILS for research & collegial
atmosphere. BEST COLLEAGUES!
Gus Friedrich for understanding & time

Relevance.ppt

More Related Content

Similar to Relevance.ppt

Recently uploaded

Relevance.ppt

Editor's Notes