Concept-Based Information Retrieval using Explicit Semantic Analysis

Concept-Based Information Retrieval using Explicit Semantic Analysis
M.Sc. Seminar talk
Ofer Egozi, CS Department, Technion
Supervisor: Prof. Shaul Markovitch
24/6/09

Information Retrieval
Query
IR
Recall
Precision

Ranked retrieval
Query
IR

Keyword-based retrieval
Bag Of Words (BOW)
Query
IR

Problem: retrieval misses
TREC document LA071689-0089
“ANCIENT ARTIFACTS FOUND. Divers have recovered artifacts lying underwater for more than 2,000 years in the wreck of a Roman ship that sank in the Gulf of Baratti, 12 miles off the island of Elba, newspapers reported Saturday."
TREC topic #411
salvaging shipwreck treasure
I
I
Query
IR

The vocabulary problem
Identity: Syntax
(tokenization, stemming…)
Similarity: Synonyms (Wordnet etc.)
Relatedness: Semantics / world knowledge
(???)
?
[but also shipping/treasurer]
Synonymy / Polysemy
?
[but also deliver/scavenge/relieve]

Concept-based retrieval
IR

Concept-based representations
Human-edited Thesauri (e.g. WordNet)
Source: editors , concepts: words, mapping: manual
Corpus-based Thesauri (e.g. co-occurrence)
Source: corpus , concepts: words , mapping: automatic
Ontology mapping (e.g. KeyConcept)
Source: ontology , concepts: ontology node(s) , mapping: automatic
Latent analysis (e.g. LSA, pLSA, LDA)
Source: corpus , concepts: word distributions , mapping: automatic
Insufficient granularity
Non-intuitive Concepts
Expensive repetitive computations
Non-scalable solution

Concept-based representations
Human-edited Thesauri (e.g. WordNet)
Source: editors , concepts: words, mapping: manual
Corpus-based Thesauri (e.g. co-occurrence)
Source: corpus , concepts: words , mapping: automatic
Ontology mapping (e.g. KeyConcept)
Source: ontology , concepts: ontology node(s) , mapping: automatic
Latent analysis (e.g. LSA, pLSA, LDA)
Source: corpus , concepts: word distributions , mapping: automatic
Is it possible to devise a
concept-based representation, that is scalable, computationally
feasible, and uses intuitive
and granular concepts?
Insufficient granularity
Non-intuitive Concepts
Expensive repetitive computations
Non-scalable solution

Explicit Semantic Analysis
Gabrilovich and Markovitch (2005,2006,2007)

Explicit Semantic Analysis (ESA)
Wikipedia is viewed as an ontology - a collection of ~1M concepts
World War II
Panthera
Jane Fonda
Island
concept

Every Wikipedia article represents a concept
Panthera
Cat [0.92]
Leopard [0.84]
Article words are associated with the concept(TF.IDF)
Roar [0.77]
concept

Panthera
Cat [0.92]
Leopard [0.84]
Roar [0.77]

Panthera
The semantics of a word is the vector of its associations with Wikipedia concepts
Cat [0.92]
Leopard [0.84]
Panthera
[0.92]
Cat
[0.95]
Jane Fonda
[0.07]
Cat
Roar [0.77]

The semantics of a text fragment is the average vector (centroid) of the semantics of its words
In practice – disambiguation…
Mouse (computing)
[0.81]
MickeyMouse[0.81]
Game Controller
[0.64]
Button
[0.93]
Game Controller
[0.32]
Mouse (rodent)
[0.91]
John Steinbeck
[0.17]
Mouse (computing)
[0.95]
Mouse (rodent)
[0.56]
Dick Button
[0.84]
Mouse (computing)
[0.84]
Drag- and-drop
[0.91]
button
mouse
mouse button
mouse button

MORAG*: An ESA-based information retrieval algorithm
*MORAG: Flail in Hebrew
“Concept-based feature generation and selection for information retrieval”, AAAI-2008

Enrich documents/queries
ESA
IR
Query
Constraint: use only the
strongest concepts

Problem: documents (in)coherence
TREC document LA120790-0036
REFERENCE BOOKS SPEAK VOLUMES TO KIDS;
With the school year in high gear, it's a good time to consider new additions to children's home reference libraries…
…Also new from Pharos-World Almanac: "The World Almanac InfoPedia," a single-volume visual encyclopedia designed for ages 8 to 16…
…"The Doubleday Children's Encyclopedia," designed for youngsters 7 to 11, bridges the gap between single-subject picture books and formal reference books…
…"The Lost Wreck of the Isis" by Robert Ballard is the latest adventure in the Time Quest Series from Scholastic-Madison Press ($15.95 hardcover). Designed for children 8 to 12, it tells the story of Ballard's 1988 discovery of an ancient Roman shipwreck deep in the Mediterranean Sea…
Document is judged relevant for topic 411 due to one relevant passage in it
Not an issue in BOW retrieval where words are indexed independently. How to deal with in concept-based?
Concepts generated for this document will average to the books / children concepts, and lose the shipwreck mentions…

Solution: split to passages
ESA
IR
Query
ConceptScore(d) =
ConceptScore(full-doc) +
max ConceptScore(passage)
passaged
Index both full document and passages.
Best performance achieved by fixed-length overlapping sliding windows.

Morag ranking
Score(q,d) =
ConceptScore(q,d) +
(1-)KeywordScore(q,d)
IR
Query

ESA-based retrieval example

Shipwreck
Treasure
Maritime archaeology
Marine salvage
History of the British Virgin Islands
Wrecking (shipwreck)
Key West, Florida
Flotsam and jetsam
Wreck diving
Spanish treasure fleet



Scuba diving
Wreck diving
RMS Titanic
USS Hoel (DD-533)
Shipwreck
Underwater archaeology
USS Maine (ACR-1)
Maritime archaeology
Tomb Raider II
USS Meade (DD-602)

Problem: irrelevant docs retrieved

Estonia
Economy of Estonia
Estonia at the 2000 Summer Olympics
Estonia national football team
Estonia at the 2006 Winter Olympics
Baltic Sea
Eurozone
TiitVähi
Military of Estonia

TREC topic #434
!
I
I
Estonia economy
“Olympic News In Brief: Cycling win for Estonia.
Erika Salumae won Estonia's first Olympic gold when retaining the women's cycling individual sprint title she won four years ago in Seoul as a Soviet athlete. "

??
??

2006 Commonwealth Games
1992 Summer Olympics
Athletics at the 2004 Summer Olympics
2006 Winter Olympics
Cross-country skiing 2006 Winter Olympics
New Zealand at the 2006 Winter Olympics

“Economy” is not mentioned, but TF·IDF of “Estonia” is strong enough to trigger this concept on its own…

Selection could remove noisy ESA concepts
However, IR task provides no training data…
Problem: selecting query features
Focus on query concepts - Query is short and noisy, while FS at indexing lacks context
Utility function U(+|-) requires target measure >> training set
U
f
=ESA(q)
Filter
f’

Solution: Pseudo Relevance Feedback
Use BOW results as positive / negative examples

ESA feature selection methods
IG (filter) – calculate each feature’s Information Gain in separating positive and negative examples, take best performing features
RV (filter) – add concepts in the positive examples to candidate features, and re-weight all features based on their weights in examples
IIG (wrapper) – find subset of features that best separates positive and negative examples, employing heuristic search

Estonia
Economy of Estonia
Estonia national football team
Baltic Sea
Eurozone
TiitVähi
Military of Estonia

ESA-based retrieval – FS example

Monetary Policy
Euro
Economy of Europe
Nordic Countries
Prime Minister of Estonia

Estonia economy
“Olympic News In Brief: Cycling win for Estonia.
Erika Salumae won Estonia's first Olympic gold when retaining the women's cycling individual sprint title she won four years ago in Seoul as a Soviet athlete. "





Broad
features
Noise
features
RV adds
features
Useful ones “bubble up”





Neoliberalism
2006 Commonwealth Games
Athletics at the 2004 Summer Olympics
2006 Winter Olympics
Cross-country skiing 2006 Winter Olympics
New Zealand at the 2006 Winter Olympics

Morag evaluation
Testing over TREC-8 and Robust-04 datasets (528K documents, 50 web-like queries)
Feature selection is highly effective

Morag evaluation
Significant performance improvement, over our own baseline and also over top performing TREC-8 BOW baselines
Concept-based performance by itself is quite low, a major reason is the TREC ‘pooling’ method, which implies that relevant documents found only by Morag will not be judged as such…

Morag evaluation
Optimal (“Oracle”) selection analysis shows much more potential for Morag

Morag evaluation
Pseudo-relevance proves to be a good approximation of actual relevance

Conclusion
Morag: a new methodology for concept-based information retrieval
Documents and query are enhanced by Wikipedia concepts
Informative features are selected using pseudo-relevance feedback
The generated features improve the performance of BOW-based systems

Thank you!