Reconsidering Relevance Daniel Tunkelang Chief Scientist, Endeca

howdy! 1988 – 1992 1993 – 1998 1999 -

1988 – 1992

1993 – 1998

1999 -

overview what is relevance? what’s wrong with relevance? what are the alternatives?

what is relevance?

what’s wrong with relevance?

what are the alternatives?

but first let’s set the stage

iconic businesses of the 20 th and 21 st centuries I’m Feeling Lucky

process and scale orchestration

but there’s a dark side

users are satisfied

an interesting contrast “ Search on the internet is solved. I always find what I need. But why not in the enterprise? Seems like a solution waiting to happen.” - a Fortune 500 CTO

“ Search on the internet is solved. I always find what I need. But why not in the enterprise? Seems like a solution waiting to happen.”

- a Fortune 500 CTO

the real questions What is “search on the internet” and why is it perceived a solved problem? What is “search in the enterprise” and why is it perceived as an unsolved problem? And what does this have to do with relevance?

What is “search on the internet” and why is it perceived a solved problem?

What is “search in the enterprise” and why is it perceived as an unsolved problem?

And what does this have to do with relevance?

easy vs. hard search problems easy where to buy Ender in Exile ? hard good novel to read on the beach? easy proof that sorting has n log n lower bound? hard algorithm to sort partially ordered set, given a constant-time comparator?

easy where to buy Ender in Exile ?

hard good novel to read on the beach?

easy proof that sorting has n log n lower bound?

hard algorithm to sort partially ordered set, given a constant-time comparator?

what is relevance? what’s wrong with relevance? what are the alternatives?

what is relevance?

what’s wrong with relevance?

what are the alternatives?

defining relevance Relevance is defined as a measure of information conveyed by a document relative to a query. It is shown that the relationship between the document and the query, though necessary, is not sufficient to determine relevance. William Goffman, On relevance as a measure, 1964.

Relevance is defined as a measure of information conveyed by a document relative to a query. It is shown that the relationship between the document and the query, though necessary, is not sufficient to determine relevance.

William Goffman, On relevance as a measure, 1964.

we need more definitions

let’s work top-down information retrieval (IR) = study of retrieval of information (not data) from collection of written documents retrieved documents aim at satisfying user information need

information retrieval (IR) = study of retrieval of information (not data) from collection of written documents retrieved documents aim at satisfying user information need

IR assumes information needs user information need = natural language declaration of informational need of user query = expression of user information need in input language provided by information system

user information need = natural language declaration of informational need of user

query = expression of user information need in input language provided by information system

relevance drives IR modeling modeling = studies algorithms used for ranking documents according to system assigned likelihood of relevance model = a set of premises and an algorithm for ranking documents with regard to a user query

modeling = studies algorithms used for ranking documents according to system assigned likelihood of relevance

model = a set of premises and an algorithm for ranking documents with regard to a user query

a relevance-centric approach information Need query select from results rank using IR model USER: SYSTEM: tf-idf PageRank

what is relevance? what’s wrong with relevance? what are the alternatives?

what is relevance?

what’s wrong with relevance?

what are the alternatives?

our first communication problem information need query 2 words? natural language? telepathy?

2 words?

natural language?

telepathy?

and the game of telephone continues query rank using IR model cumulative error relevance is subjective what Goffman said

cumulative error

relevance is subjective

what Goffman said

and hopefully users feel lucky rank using IR model selection bias inefficient channel backup plan? select from results

selection bias

inefficient channel

backup plan?

queries are misinterpreted Results 1-10 out of about 344,000,000 for ir

ranked lists are inefficient

assumptions of relevance-centric approach self-awareness self-expression model knows best answer is a document one-shot query

self-awareness

self-expression

model knows best

answer is a document

one-shot query

can we do better?

what is relevance? what’s wrong with relevance? what are the alternatives?

what is relevance?

what’s wrong with relevance?

what are the alternatives?

human-computer information retrieval don’t just guess the user’s intent optimize communication increase user responsibility and control require and reward human intellectual effort “ Toward Human-Computer Information Retrieval” Gary Marchionini

don’t just guess the user’s intent

optimize communication

increase user responsibility and control

require and reward human intellectual effort

human computer information retrieval

a concrete use case Colleague: Hey Daniel! You should check out what this guy Steve Pollitt’s been researching. Sounds right up your alley. Daniel: Sure thing, I’ll look into it.

Colleague: Hey Daniel! You should check out what this guy Steve Pollitt’s been researching. Sounds right up your alley.

Daniel: Sure thing, I’ll look into it.

google him!

google scholar him?

rexa him?

getting better

hcir-inspired interface

tags provide summarization and guidance

my information need evolves as i learn

hcir – implementing the vision

scatter/gather: a search for “star”

faceted search

practical considerations which facets to show which facet values to show when to suggest faceted refinement how to automate faceted classification

which facets to show

which facet values to show

when to suggest faceted refinement

how to automate faceted classification

showing the right facets: microwaves

showing the right facets: ceiling fans

query-driven clarification before refinement Matching Categories include: Appliances > Small Appliances > Irons & Steamers Appliances > Small Appliances > Microwaves & Steamers Bath > Sauna & Spas > Steamers Kitchen > Bakeware & Cookware > Cookware > Open Stock Pots > Double Boilers & Steamers Kitchen > Small Appliances > Steamers

Matching Categories include:

Appliances > Small Appliances > Irons & Steamers

Appliances > Small Appliances > Microwaves & Steamers

Bath > Sauna & Spas > Steamers

Kitchen > Bakeware & Cookware > Cookware >

Open Stock Pots > Double Boilers & Steamers

Kitchen > Small Appliances > Steamers

results-driven clarification before refinement Search : storage

crowd-sourcing to tag documents

hcir cheats the precision / recall trade-off recall precision

set retrieval 2.0 set retrieval that responds to queries with overview of the user's current context organized set of options for exploration contextual summaries of document sets optimize system’s communication with user query refinement options optimize user’s communication with system

set retrieval that responds to queries with

overview of the user's current context

organized set of options for exploration

contextual summaries of document sets

optimize system’s communication with user

query refinement options

optimize user’s communication with system

hcir using set retrieval 2.0 emphasize set summaries over ranked lists establish a dialog between the user and the data enable exploration and discovery

emphasize set summaries over ranked lists

establish a dialog between the user and the data

enable exploration and discovery

think outside the (search) box relevance-centric search solves many use cases but not some of the most valuable ones support interaction, exploration human-computer information retrieval

relevance-centric search solves many use cases

but not some of the most valuable ones

support interaction, exploration

human-computer information retrieval

one more thing …

“ Google's mission is to organize the world's information and make it universally accessible and useful.”

organizer or referee?

thank you communication 1.0 email: [email_address] communication 2.0 blog: http://thenoisychannel.com twitter: http://twitter.com/dtunkelang

communication 1.0

email: [email_address]

communication 2.0

blog: http://thenoisychannel.com

twitter: http://twitter.com/dtunkelang