Enabling Exploration Through Text Analytics
Loading...
Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.
3 Favorites
-
Mahesh CR, Senior Manager at Thomson Reuters,
favorited this 5 months ago
-
Hilary Mason, Computer Scientist at Path101, favorited this 5 months ago
-
Daniel Tunkelang, Noogler at Google, favorited this 5 months ago
Enabling Exploration Through Text Analytics - Presentation Transcript
- Enabling Exploration through Text Analytics Daniel Tunkelang Chief Scientist, Endeca
- overview
- information seeking tools
- need to support exploration
- text analytics can help
- you can do this here and now
- real-world information seeking examples
- looking for health information
- looking for work-related information
- reminder
- search and text analytics
- are a means, not an end
- example 1: looking for health information
- six months into my wife’s pregnancy, we
- discovered that she had gestational diabetes
- how to learn more?
- google: the default option for most
- in government we trust: fda.gov
- maybe the private sector knows best: webmd powered by
- success – and a sticky site powered by
- example 2: looking for work-related information
- need to ramp up summer
- interns on text mining
- how to find a good book?
- let’s try google again
- google: the gateway to wikipedia?
- the library of congress (loc.gov)
- triangle research libraries: next-gen catalog powered by
- faceted search enables query refinement powered by
- take-away #1
- exploratory search support:
- a must-have for many information needs
- text analytics
- categorization
- named entity detection
- term extraction
- sentiment analysis
- vague term, lots of see-alsos
- text mining
- information extraction
- content enrichment
- newssift: text analytics enabling exploration powered by categorization named entity detection term extraction sentiment analysis
- exploring the news about facebook powered by
- facebook: the good powered by Social Utility Iphone Application
- facebook: the bad powered by Criminal Behavior Litigation And Settlement
- take-away #2
- text analytics enable
- exploratory search
- text analytics is here and now ? ? ?
- lots of off-the-shelf options and more!
- caveats
- rule-based techniques are domain-specific
- statistical techniques rely on trained models
- plan for errors, inconsistency
- document vs. corpus analysis
- problems with entity extraction
- moderate precision, but low recall
- not just noisy, but inconsistent
- corpus analysis can help!
- look for ways to cheat! recall precision
- division of labor people supply vocabulary machine annotates documents http://www.precolumbianwomen.com/images/inca-labor.10.gif
- example: ACM digital library
- opportunity
- repository of (sometimes) author-tagged documents
- high-precision tags: very few false positives
- challenge
- poor reuse of vocabulary: most tags unique
- low-recall tags: 90% false negatives
- as is, tags were not useful for exploration
- solution
- bootstrap on author-supplied tags
- prune 600K+ tags to 10K by
- imposing frequency threshold
- normalizing by case and singular/plural
- eliminating infrequent subphrases
- mine documents using resulting vocabulary
- manually validate most frequently assigned tags
- example: a search for boeing powered by
- it’s a HITS!
- if you prefer sports to computer science
- no author-supplied tags
- use search logs instead
- supplement with authority files
- team names
- player names
- mine documents using resulting vocabulary
- roger clemens, then and now powered by
- pivoting to a different view powered by
- take-away #3
- this is not vapor ware;
- text analytics to enable exploration
- is available here and now
- looking forward
- better tags are the beginning, not the end
- improve with manual and automatic processing
- give users control over precision / recall trade-off
- help users and content creators help you
- in closing
- exploratory search = must-have, not nice-to-have
- text analytics are a key enabler
- the technology is real, here, and now
- thank you…and come to SIGIR!
- communication 1.0
- email: [email_address]
- communication 2.0
- blog: http://thenoisychannel.com
- twitter: http://twitter.com/dtunkelang
- SIGIR: July 19-23 in Boston
- Industry Track on July 22 nd !
+
1045 views, 3 favs, more stats
Enterprises are awash in textual documents that rep more
More info about this document
© All Rights Reserved
Go to text version
- Total Views 1045
- 883 on SlideShare
- Comments 0
- Favorites 3
- Downloads 32
Most viewed embeds
All embeds
- Upload Info
- Also on LinkedIn
- Uploaded via SlideShare
- Uploaded as Microsoft PowerPoint
0 comments
Post a comment