Google Tech Talk: Reconsidering Relevance

Reconsidering Relevance Daniel Tunkelang Chief Scientist, Endeca

howdy!
1988 – 1992
1993 – 1998
1999 -

overview
what is relevance?
what’s wrong with relevance?
what are the alternatives?

but first let’s set the stage

iconic businesses of the 20 th and 21 st centuries I’m Feeling Lucky

process and scale orchestration

but there’s a dark side

users are satisfied

an interesting contrast
“ Search on the internet is solved. I always find what I need. But why not in the enterprise? Seems like a solution waiting to happen.”
- a Fortune 500 CTO

the real questions
What is “search on the internet” and why is it perceived a solved problem?
What is “search in the enterprise” and why is it perceived as an unsolved problem?
And what does this have to do with relevance?

easy vs. hard search problems
easy where to buy Ender in Exile ?
hard good novel to read on the beach?
easy proof that sorting has n log n lower bound?
hard algorithm to sort partially ordered set, given a constant-time comparator?

what is relevance?

defining relevance
Relevance is defined as a measure of information conveyed by a document relative to a query. It is shown that the relationship between the document and the query, though necessary, is not sufficient to determine relevance.
William Goffman, On relevance as a measure, 1964.

we need more definitions

let’s work top-down
information retrieval (IR) = study of retrieval of information (not data) from collection of written documents retrieved documents aim at satisfying user information need

IR assumes information needs
user information need = natural language declaration of informational need of user
query = expression of user information need in input language provided by information system

relevance drives IR modeling
modeling = studies algorithms used for ranking documents according to system assigned likelihood of relevance
model = a set of premises and an algorithm for ranking documents with regard to a user query

a relevance-centric approach information Need query select from results rank using IR model USER: SYSTEM: tf-idf PageRank

what is relevance?

our first communication problem information need query
2 words?
natural language?
telepathy?

and the game of telephone continues query rank using IR model
cumulative error
relevance is subjective
what Goffman said

and hopefully users feel lucky rank using IR model
selection bias
inefficient channel
backup plan?
select from results

queries are misinterpreted Results 1-10 out of about 344,000,000 for ir

ranked lists are inefficient

assumptions of relevance-centric approach
self-awareness
self-expression
model knows best
answer is a document
one-shot query

can we do better?

what is relevance?

human-computer information retrieval
don’t just guess the user’s intent
optimize communication
increase user responsibility and control
require and reward human intellectual effort
“ Toward Human-Computer Information Retrieval” Gary Marchionini

human computer information retrieval

a concrete use case
Colleague: Hey Daniel! You should check out what this guy Steve Pollitt’s been researching. Sounds right up your alley.
Daniel: Sure thing, I’ll look into it.

google him!

google scholar him?

rexa him?

getting better

hcir-inspired interface

tags provide summarization and guidance

my information need evolves as i learn

hcir – implementing the vision

scatter/gather: a search for “star”

faceted search

practical considerations
which facets to show
which facet values to show
when to suggest faceted refinement
how to automate faceted classification

showing the right facets: microwaves

showing the right facets: ceiling fans

query-driven clarification before refinement
Matching Categories include:
Appliances > Small Appliances > Irons & Steamers
Appliances > Small Appliances > Microwaves & Steamers
Bath > Sauna & Spas > Steamers
Kitchen > Bakeware & Cookware > Cookware >
Open Stock Pots > Double Boilers & Steamers
Kitchen > Small Appliances > Steamers

results-driven clarification before refinement Search : storage

crowd-sourcing to tag documents

hcir cheats the precision / recall trade-off recall precision

set retrieval 2.0
set retrieval that responds to queries with
overview of the user's current context
organized set of options for exploration
contextual summaries of document sets
optimize system’s communication with user
query refinement options
optimize user’s communication with system

hcir using set retrieval 2.0
emphasize set summaries over ranked lists
establish a dialog between the user and the data
enable exploration and discovery

think outside the (search) box
relevance-centric search solves many use cases
but not some of the most valuable ones
support interaction, exploration
human-computer information retrieval

one more thing …

“ Google's mission is to organize the world's information and make it universally accessible and useful.”

organizer or referee?

thank you
communication 1.0
email: [email_address]
communication 2.0
blog: http://thenoisychannel.com
twitter: http://twitter.com/dtunkelang

http://thenoisychannel.com	372
http://blogs.oracle.com	282
http://www.cs.cmu.edu	185
http://tantrieuf31.blogspot.com	57
http://catacroquer.blogspot.com	28
http://www.techgig.com	27
http://vielmetti.typepad.com	16
http://www.slideshare.net	11
https://blogs.oracle.com	6
http://www.agglom.com	4
http://translate.googleusercontent.com	4
http://skydivedadpublishing.com	3
http://sambalewa.web.id	2
http://gmsvmd48	2
http://static.slideshare.net	1
http://www.bloggen.be	1
http://smartdatacollective.com	1
http://catacroquer.blogspot.mx	1
http://www.javaoracleblog.com	1
http://catacroquer.blogspot.com.es	1

Google Tech Talk: Reconsidering Relevance

by Daniel Tunkelang on Jan 08, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

20 Embeds 1,005

Statistics

Google Tech Talk: Reconsidering Relevance — Presentation Transcript