Semantic Text Processing Powered by Wikipedia Maxim Grinev [email_address]

Technology Overview Next Generation Text Analysis bootstrapped by Wikipedia Wikipedia is a new enabling resource for NLP Comprehensive coverage ( 6M terms versus 65K in Britannica ) Continuously brought up-to-date Rich Structure ( cross-references between articles, categories, redirect pages, disambiguation pages, info-boxes ) New Algorithms: Advanced NLP: Word Sense Disambiguation, Keywords Extraction, Topic Inference Automatic Ontology Management: Organizing Concept into Thematically Grouped Tag Clouds Semantic Search: Concept-based Similarity Search, Smart Faceted Navigation Improved Recommendations: Semantic Document Similarity Zero-cost deployment and customization: No machine learning techniques which require human labor, no “cold start”

Next Generation Text Analysis bootstrapped by Wikipedia

Wikipedia is a new enabling resource for NLP

Comprehensive coverage ( 6M terms versus 65K in Britannica )

Continuously brought up-to-date

Rich Structure ( cross-references between articles, categories, redirect pages, disambiguation pages, info-boxes )

New Algorithms:

Advanced NLP: Word Sense Disambiguation, Keywords Extraction, Topic Inference

Automatic Ontology Management: Organizing Concept into Thematically Grouped Tag Clouds

Semantic Search: Concept-based Similarity Search, Smart Faceted Navigation

Improved Recommendations: Semantic Document Similarity

Zero-cost deployment and customization: No machine learning techniques which require human labor, no “cold start”

We analyse Wikipedia Links Structure to compute Semantic Relatedness of Wikipedia terms We use Dice-measure with weighted links (bi-directional links, direct links, “see also” links, etc) Basic Technique: Semantic Relatedness of Terms Dmitry Lizorkin, Pavel Velikhov, Maxim Grinev, Denis Turdakov Accuracy Estimate and Optimization Techniques for SimRank Computation, VLDB 2008

We analyse Wikipedia Links Structure to compute Semantic Relatedness of Wikipedia terms

We use Dice-measure with weighted links (bi-directional links, direct links, “see also” links, etc)

Terms Detection and Disambiguation Example: IBM may stand for International Business Machines Corp . or International Brotherhood of Magicians We use Wikipedia redirection (synonyms) and disambiguation pages (homonyms) to detect and disambiguate terms in a text Example: Platform is mentioned in the context of implementation , open-source , web-server, HTTP Denis Turdakov, Pavel Velikhov “ Semantic Relatedness Metric for Wikipedia Concepts Based on Link Analysis and its Application to Word Sense Disambiguation ” SYRCoDIS, 2008

Example: IBM may stand for International Business Machines Corp . or International Brotherhood of Magicians

We use Wikipedia redirection (synonyms) and disambiguation pages (homonyms) to detect and disambiguate terms in a text

Example: Platform is mentioned in the context of implementation , open-source , web-server, HTTP

Denis Turdakov, Pavel Velikhov

“ Semantic Relatedness Metric for Wikipedia Concepts Based on

Link Analysis and its Application to Word Sense Disambiguation ”

SYRCoDIS, 2008

Keywords Extraction Build document semantic graph using semantic relatedness between Wikipedia terms detected in the doc Discover community structure of the document semantic graph Community – densely interconnected group of nodes in a graph Girvan-Newman algorithm for detection community structure in networks Select “best” communities: Densed communities contain key terms Sparse communities contain not important terms, and possible disambiguation mistakes Maria Grineva, Maxim Grinev, Dmitry Lizorkin Extracting Key Terms From Noisy and Multitheme Documents WWW2009: 18th International World Wide Web Conference

Build document semantic graph using semantic relatedness between Wikipedia terms detected in the doc

Discover community structure of the document semantic graph

Community – densely interconnected group of nodes in a graph

Girvan-Newman algorithm for detection community structure in networks

Select “best” communities:

Densed communities contain key terms

Sparse communities contain not important terms, and possible disambiguation mistakes

Keywords Extraction (Example) Semantic graph built from a news article " Apple to Make ITunes More Accessible For the Blind "

Advantages of the Keywords Extraction Method No training . Instead of training the system with hand-created examples, we use semantic information derived from Wikipedia Noise and multi-theme stability. Good at filtering out noise and discover topics in Web pages Thematically grouped key terms . Significantly improve further inferring of document topics High accuracy . Evaluated using human judgments

No training . Instead of training the system with hand-created examples, we use semantic information derived from Wikipedia

Noise and multi-theme stability. Good at filtering out noise and discover topics in Web pages

Thematically grouped key terms . Significantly improve further inferring of document topics

High accuracy . Evaluated using human judgments

Other Methods General Topic Inference for a doc using spreading activation over Wikipedia categories graph Example: Amazon EC2, Microsoft Azure, Google MapReduce => Cloud Computing Building Thematically Grouped Tag Clouds for many docs Girvan-Newman algorithm to split into thematic groups Topic inference for each group Document classification Semantic similarity is used to indentify indirect relationships between terms (e.g. a doc about collaborative filtering is classified to recommender system )

General Topic Inference for a doc

using spreading activation over Wikipedia categories graph

Example: Amazon EC2, Microsoft Azure, Google MapReduce => Cloud Computing

Building Thematically Grouped Tag Clouds for many docs

Girvan-Newman algorithm to split into thematic groups

Topic inference for each group

Document classification

Semantic similarity is used to indentify indirect relationships between terms (e.g. a doc about collaborative filtering is classified to recommender system )

Semantic Search & Navigation Search by Concept : Advantages of query and in-doc terms disambiguation Result: documents about the concept and related concepts ordered by relevance (keywordness) Smart Faceted Navigation : query-relevant facets using semantic relatedness Concept-tips to grasp the result documents Each document in the result is accompanied with concepts-tips that explain how this document is relevant to the Query

Search by Concept :

Advantages of query and in-doc terms disambiguation

Result: documents about the concept and related concepts ordered by relevance (keywordness)

Smart Faceted Navigation : query-relevant facets using semantic relatedness

Concept-tips to grasp the result documents

Each document in the result is accompanied with concepts-tips that explain how this document is relevant to the Query

Facets Generation

Facets Generation (cont.)

Facets Generation (cont.)

Facets Generation (cont.)

Thank You!