Unsupervised Word Sense Discrimination By Clustering Similar Contexts Ted Pedersen University of Minnesota, Duluth http:// www.d.umn.edu/~tpederse Research Supported by National Science Foundation Faculty Early Career Development Award (#0092784)
Bridget McInnes (MS 2004) -> Univ of Minn/TC (PhD)
Siddharth Patwardhan (MS 2003) -> Univ of Utah (PhD)
Saif Mohammad (MS 2003) -> Univ of Toronto (PhD)
Supervised word sense disambiguation, sense tagged data
Satanjeev Banerjee (MS 2002) -> CMU (PhD)
Ngram Statistics Package , WordNet::Similarity
I do research…more about that soon…
Natural Language Processing
Graduate NLP class worked on essay grading systems in Fall 2004
More on that later…
Operating Systems Practicum
Overall Research Objectives
Assign meanings to words
Bank means Financial Institution
Group words according to meaning
Line, Cord, Cable are synonyms
Organize texts according to content
Records of patients with similar ailments
Organize concepts by relationships
Rachel is a friend of Ross
Making Free Software Mostly Perl, All CopyLeft
Identify similar contexts
Ngram Statistics Package
Identify interesting sequences of words
Measure similarity among concepts
All words sense disambiguation
Find sets of related words
SyntaLex and Duluth systems
Unsupervised Word Sense Discrimination By Clustering Similar Contexts With Considerable Assistance From Anagha Kulkarni (M.S. 2006) Amruta Purandare (M.S. 2004)
Overview shells exploded in a US diplomatic complex in Liberia shell scripts are user interactive artillery guns were used to fire highly explosive shells the biggest shop on the shore for serious shell collectors shell script is a series of commands written into a file that Unix executes she sells sea shells by the sea shore sherry enjoys walking along the beach and collecting shells firework shells exploded onto usually dark screens in a variety of colors shells automate system administrative tasks we specialize in low priced corals, starfish and shells we help people in identifying wonderful sea shells along the coastlines shop at the biggest shell store by the shore shell script is much like the ms dos batch file
sherry enjoys walking along the beach and collecting shells we specialize in low priced corals, starfish and shells we help people in identifying wonderful sea shells along the coastlines shop at the biggest shell store by the shore she sells sea shells by the sea shore the biggest shop on the shore for serious shell collectors shell script is much like the ms dos batch file shell script is a series of commands written into a file that Unix executes shell scripts are user interactive shells automate system administrative tasks shells exploded in a US diplomatic complex in Liberia firework shells exploded onto usually dark screens in a variety of colors artillery guns were used to fire highly explosive shells
Dictionaries are fixed and static, relative to the world at least
Sense distinctions made in dictionaries are not always the right ones for NLP applications.
29 senses of line?
Dictionaries don’t agree.
So which one do you use?
Identify contexts that use a word in similar way.
I drove my car to the house.
My car doesn’t drive very well any more.
Assume that word has similar or related meanings.
Automatically create a descriptive label that serves as a definition of that word in those contexts .
… make it possible to automatically discover meanings and categorize words relative to them without the use of difficult to create and maintain resources…
Pectens or Scallops are one of the few bivalve shells that actually swim. This is accomplished by rapidly opening & closing their valves, sending the shell backward.
Fire marshals hauled out something that looked like a rifle with tubes attached to it, along with several bags of bullets and shells .
If you hear a snapping sound when you’re in the water, chances are it is the sound of the valves hitting together as it opens and shuts its shell .
Teenagers tried to make a bomb or some kind of homemade fireworks by taking the bullets and shotgun shells apart and collecting the black powder.
Bivalve shells are mollusks with two valves joined by a hinge. Most of the 20,000 species are marine including clams, mussels, oysters and scallops.
There was an explosion in one of the shells , it flamed over the top of the other shells and sealed in the fireworks, so when they ignited, it made it react like a pipe bomb."
These edible oysters are the most commonly known throughout the world as a popular source of seafood. The shell is porcelaneous and the pearls produced from these edible oysters have little value.
Surface Lexical Features
in today’s world the scallop is a popular design in architecture and is well known as the shell gasoline logo if you hear a snapping sound when you’re in the water chances are it is the sound of the valves hitting together as it opens and shuts its shell
she sells sea shells on the sea shore
the <>sea on <> the sea<>shore shells<> on sea<>shells she <>sells sells<>sea Rejected Selected
Experiment 1: Results POS wise 29 NOUNS 28verbs 15 adjs No of words of a POS for which experiment obtained accuracy more than Majority 8 7 3 5 7 6 MAT COS SOC UNI BI MAT COS MAT COS 0 1 0 0 1 1 9 13 5 5 6 11 SOC UNI BI SOC UNI BI
Experiment 1: Results Feature wise SOC BI UNI 32 18 38 1 1 6 11 7 6 MAT COS N ADJ V MAT COS MAT COS 0 1 9 13 8 7 0 0 5 5 3 5 N ADJ V N ADJ V
Experiment 1: Results Measure wise COS MAT 49 39 0 5 5 1 1 13 11 7 6 BI UNI SOC N ADJ V BI UNI SOC 0 5 3 0 1 9 6 8 7 N ADJ V
Experiment 1: Conclusions
Scaling done by Cosine helps
1 st order contexts very sparse
Similarity space even more sparse
Experiment 2: 2 nd Order Contexts and RBR
SC1 with Bi-gram Matrix
PB1 with Bi-gram Features
UPGMA, Similarity Space
RB, Vector Space
Co-occurrence Matrix, SVD
RB, Vector Space
UPGMA, Similarity Space
Schütze (2 nd Order Contexts) Pedersen & Bruce (1 st Order Contexts)
Experiment 2: Sval2 Results Bi-grams Vs Co-occurrences Bi-gram = COC 0 2 1 Bi-gram < COC 1 1 4 Bi-gram > COC 3 3 9 Bi-gram = COC 0 1 1 Bi-gram < COC 2 4 6 Bi-gram > COC 2 1 7 V A N PB1 Vs PB3 SC1 Vs SC3
Experiment 2: Sval2 Results RB Vs UPGMA RB = UPGMA 1 0 4 RB < UPGMA 0 5 2 RB > UPGMA 3 1 8 RB = UPGMA 1 2 1 RB < UPGMA 2 0 4 RB > UPGMA 1 4 9 V A N PB1 Vs PB2 SC1 Vs SC2
Experiment 2: Sval2 Results Comparing with MAJ 5 2 0 3 PB3 > MAJ 6 1 1 4 PB1 > MAJ 9 2 1 6 SC2 > MAJ 9 0 2 7 PB2 > MAJ 10 2 2 6 SC1 > MAJ 12 1 3 8 SC3 > MAJ Total V A N