Text Analytics: Yesterday, Today and Tomorrow

Michael Ferretti Tony Russell-Rose Vladimir Zelevinsky Text Analytics: Yesterday Today Tomorrow

Part 1 (of 3)
WHAT?

What is Text Analytics?
A set of linguistic, analytical and predictive techniques to extract structure and meaning extracted from unstructured documents
Text Analytics ~= Natural Language Processing ~= Text Mining
Text Mining -> Scientific / technical context, automated processing
Text Analytics -> Business context, interactive apps
Copyright ©2010 Endeca Technologies, Inc. All rights reserved. Proprietary and confidential. “ NLP ” vs. “ Text Analytics ”

Why is Text Analytics Important?
‘ 80% of corporate information is unstructured’
Entire value chain for some organisation (media / publishing etc.)
Retail / eCommerce: Product reviews
User generated content: blogs, forums, wikis
Voice of the Customer: social media + sentiment analysis
161 billion gigabytes of digital information in 2006
approximately 988 exabytes by 2010
Audio / video still needs summaries & tags etc.
Copyright ©2010 Endeca Technologies, Inc. All rights reserved. Proprietary and confidential.

How Difficult Can It Be?
As humans we do it effortlessly ... don’t we?
DRUNK GETS NINE YEARS IN VIOLIN CASE
PROSTITUTES APPEAL TO POPE
STOLEN PAINTING FOUND BY TREE
RED TAPE HOLDS UP NEW BRIDGE
DEER KILL 300,000
RESIDENTS CAN DROP OFF TREES
INCLUDE CHILDREN WHEN BAKING COOKIES
MINERS REFUSE TO WORK AFTER DEATH

Some Fundamentals
Language is AMBIGUOUS
To find structure, we must remove ambiguity!
Lexical analysis (tokenisation)
The cat sat on the mat
I can’t tokenise this sentence
Morphology (term variations, prefixes, suffixes, etc.)
Computer, computing, compute, computed = comput*
Delegate = de-leg-ate (?)
Ratify = rat-ify (?)

More Fundamentals
Syntax (part of speech tagging)
Time flies like an arrow
Fruit flies like a banana
Eats shoots and leaves
Parsing (grammar)
I saw a venetian blind
I saw a blind venetian
Rugby is a game played by men with odd-shaped balls
Sentence boundary detection
Punctuation denotes the end of a sentence!
“ But not always!”, said Fred...

Named Entity Recognition/Information Extraction
Companies in New York != New companies in York
People, places, organisations ...
Increase precision
Support navigation
Facilitate translation, summarisation, speech synthesis, etc.
IE = template filling
Entities + relationships
Highly context dependent
Problems with:
Anaphora resolution
Word sense disambiguation

Question Answering
Give me answers, not documents!
Fact-finding vs. exploratory search
Yes/no questions ‘Is George W. Bush the current president of the USA?’
‘ Who’ questions ‘Who was the British Prime Minister before Margaret Thatcher?’
List questions ‘Which football teams have won the Champions League this decade?
Instruction-based questions ‘How do I cook lasagne?’
Explanation questions ‘Why did World War I start?’
Commands ‘Tell me the height of the Eiffel Tower.’
Question analysis -> document retrieval -> answer extraction

Part 2 (of 3)
HOW?

Text Analytics is Computer Science + Semantics .
Semantics is the study of meaning .
Definitions Universal flowchart:

No mind reading (yet). Have to use text .
Text approximates meaning .
Meaning is structured .
Meaning and structure CONCEPT CONCEPT CONCEPT

Synonymy :
one concept maps to different words.
Polysemy :
one word maps to different concepts.
Problems with text

Simplest structure: salient terms Many years later, as he faced the firing squad , Colonel Aureliano Buendía was to remember that distant afternoon when his father took him to discover ice . – Marquez (1962)

Typed entities People , places , organizations ; etc. Simple approach: word lists . More difficult: trained extractors (including sentiment ).

Highest clarity organizations for “baseball” Top terms World Series Teams 1987 Cardinals; Twins Cardinals; Twins 1988 Dodgers; Mets Dodgers; Athletics 1989 Athetics; Giants Athetics; Giants 1991 Braves; Twins Braves; Twins 1992 Blue Jays; Braves Blue Jays; Braves 1996 Yankees; Braves Yankees; Braves 1997 Indians; Marlins Indians; Marlins 1998 Yankees; Padres Yankees; Padres 1999 Braves; Mets ; Yankees Braves; Yankees 2000 Yankees; Mets Yankees; Mets 2001 Diamondbacks; Yankees Diamondbacks; Yankees 2003 Marlins Marlins; Yankees Fail: 1990, 1993, 1995, 2002.

Salient terms on a timeline: baseball Clarity scores for top terms for “ baseball ” search: No event in 1994!

Salient terms on a timeline: Iraq

Excellent corpus:
Research articles.
Written by humans.
Tagged by authors.
Case study: ACM But: Half the articles untagged. Tags sparse (90% of tags used once!) Synonyms abound. tags -> controlled tag vocabulary -> high-scoring salient tags

Co-occurrence :
salient terms that tend to occur together belong together.
Clusters

Clusters for disambiguation

Apple, meaning 1

Apple, meaning 2

Apple, meaning 3

Human brain is great at extracting information scent:
[word, word, word, …] -> meaning
Information Scent [ island, Indonesia ] [ code, Sun ] [ coffee, beans, brew ] -> Java

Vector model Can also rotate basis for the best representation: LSI – Salton (1983) Similarity between documents = cosine of the angle between their vectors

Semantic networks emotion ->

Custom Dimensions

Sentence structure parsing

It is said Mrs. Clinton promises new jobs will be created by her. N V V N N V A N V V V N part of speech tagging noun / verb phrase extraction sentence structure analysis anaphora resolution passive tense flipping triple filtering hierarchy generation Sentence structure parsing

Nouns by head noun: [ Mrs. + Hillary + Bill + President ] -> Clinton Verbs by hypernyms (broadening synonyms): [ say + tell + propose + suggest + declare ] -> express Hierarchy generation (also semantic network!)

Idea Navigation

Part 3 (of 3)
WHO?

For profit :
Lexalytics
Text Enrichment module
Text Enrichment with Sentiment Analysis
Alias-i
Term Discovery
Nstein
Newssift
For fun :
GATE (Sheffield University)
Open source, linguistic focus
RapidMiner (University of Dortmund / Rapid-I)
Open source community edition, data mining focus
WordNet, OpenCalais, LingPipe, NLPwiki, etc.
Text Analytics for fun and profit Copyright ©2010 Endeca Technologies, Inc. All rights reserved. Proprietary and confidential.

Market maturing & expanding: 75-200 %
Most vendors on target
Dominant markets:
CX, media/publishing, FS & insurance, intelligence, life sciences, e-discovery
Solutions still not standardized
Need for self-service tuning & configuration
Massive expansion in social media
Lightweight NLP for buzz analysis, brand monitoring, etc.
Partner ecosystem developing
Marketing services providers, platform vendors, CRM + call centre vendors, system integrators
Market Outlook: 2010 Copyright ©2010 Endeca Technologies, Inc. All rights reserved. Proprietary and confidential.

Conclusions We are all interested in the future, for that is where you and I are going to spend the rest of our lives. And remember, my friend, future events such as these will affect you in the future. – Edward Wood Jr. (1957)
What do we expect in the future?
Extraction leads to generation
Summarization
Generalization
Narratives
Inference and conflict resolution

Text analytics: what does it mean? Unstructured text isn't unstructured. There's always structure. Find the information scent. Let the users follow it. Don’t trust that one query is enough. Let the users interact. Text does not matter. Meaning does.

http://java.dzone.com	575
http://isquared.wordpress.com	486
http://architects.dzone.com	15
http://www.linkedin.com	6
http://translate.googleusercontent.com	4
http://openlinkeddatasolutions.magnify.net	3
http://www.dzone.com	2
https://isquared.wordpress.com	2
http://server.dzone.com	2
http://115.112.206.131	2
http://sql.dzone.com	1

Text Analytics: Yesterday, Today and Tomorrow

by Tony Russell-Rose

Accessibility

Categories

Upload Details

Usage Rights

11 Embeds 1,098

Statistics