Eurolan 2005 Pedersen


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Eurolan 2005 Pedersen

  1. 1. Language Independent Methods of Clustering Similar Contexts (with applications) Ted Pedersen University of Minnesota, Duluth [email_address]
  2. 2. The Problem <ul><li>A context is a short unit of text </li></ul><ul><ul><li>often a phrase to a paragraph in length, although it can be longer </li></ul></ul><ul><li>Input: N contexts </li></ul><ul><li>Output: K clusters </li></ul><ul><ul><li>Where each member of a cluster is a context that is more similar to each other than to the contexts found in other clusters </li></ul></ul>
  3. 3. Language Independent Methods <ul><li>Do not utilize syntactic information </li></ul><ul><ul><li>No parsers, part of speech taggers, etc. required </li></ul></ul><ul><li>Do not utilize dictionaries or other manually created lexical resources </li></ul><ul><li>Based on lexical features selected from corpora </li></ul><ul><li>No manually annotated data of any kind, methods are completely unsupervised in the strictest sense </li></ul><ul><li>Assumption: word segmentation can be done by looking for white spaces between strings </li></ul>
  4. 4. Outline (Tutorial) <ul><li>Background and motivations </li></ul><ul><li>Identifying lexical features </li></ul><ul><ul><li>Measures of association & tests of significance </li></ul></ul><ul><li>Context representations </li></ul><ul><ul><li>First & second order </li></ul></ul><ul><li>Dimensionality reduction </li></ul><ul><ul><li>Singular Value Decomposition </li></ul></ul><ul><li>Clustering methods </li></ul><ul><ul><li>Agglomerative & partitional techniques </li></ul></ul><ul><li>Cluster labeling </li></ul><ul><li>Evaluation techniques </li></ul><ul><ul><li>Gold standard comparisons </li></ul></ul>
  5. 5. Outline (Practical Session) <ul><li>Headed contexts </li></ul><ul><ul><li>Name Discrimination </li></ul></ul><ul><ul><li>Word Sense Discrimination </li></ul></ul><ul><ul><li>Abbreviations </li></ul></ul><ul><li>Headless contexts </li></ul><ul><ul><li>Email/Newsgroup Organization </li></ul></ul><ul><ul><li>Newspaper text </li></ul></ul><ul><li>Identifying Sets of Related Words </li></ul>
  6. 6. SenseClusters <ul><li>A package designed to cluster contexts </li></ul><ul><li>Integrates with various other tools </li></ul><ul><ul><li>Ngram Statistics Package </li></ul></ul><ul><ul><li>Cluto </li></ul></ul><ul><ul><li>SVDPACKC </li></ul></ul><ul><li> </li></ul>
  7. 7. Many thanks… <ul><li>Satanjeev (“Bano”) Banerjee (M.S., 2002) </li></ul><ul><ul><li>Founding developer of the Ngram Statistics Package (2000-2001) </li></ul></ul><ul><ul><li>Now PhD student in the Language Technology Institute at Carnegie Mellon University </li></ul></ul><ul><li>Amruta Purandare (M.S., 2004) </li></ul><ul><ul><li>Founding developer of SenseClusters (2002-2004) </li></ul></ul><ul><ul><li>Now PhD student in Intelligent Systems at the University of Pittsburgh http:// / </li></ul></ul><ul><li>Anagha Kulkarni (M.S., 2006, expected) </li></ul><ul><ul><li>Enhancing SenseClusters since Fall 2004! </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>National Science Foundation (USA) for supporting Bano, Amruta, Anagha and me (!) via CAREER award #0092784 </li></ul>
  8. 8. Practical Session <ul><li>Experiment with SenseClusters </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li>Has both a command line and web interface (above) </li></ul></ul><ul><li>Can be installed on Linux/Unix machine without too much work </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li>Has some dependencies that must be installed, so having supervisor access and/or sysadmin experience helps </li></ul></ul><ul><ul><li>Complete system (SenseClusters plus dependencies) is available on CD </li></ul></ul>
  9. 9. Background and Motivations
  10. 10. Headed and Headless Contexts <ul><li>A headed context includes a target word </li></ul><ul><ul><li>Our goal is to collect multiple contexts that mention a particular target word in order to try identify different senses of that word </li></ul></ul><ul><li>A headless context has no target word </li></ul><ul><ul><li>Our goal is to identify the contexts that are similar to each other </li></ul></ul>
  11. 11. Headed Contexts (input) <ul><li>I can hear the ocean in that shell. </li></ul><ul><li>My operating system shell is bash. </li></ul><ul><li>The shells on the shore are lovely. </li></ul><ul><li>The shell command line is flexible. </li></ul><ul><li>The oyster shell is very hard and black. </li></ul>
  12. 12. Headed Contexts (output) <ul><li>Cluster 1: </li></ul><ul><ul><li>My operating system shell is bash. </li></ul></ul><ul><ul><li>The shell command line is flexible. </li></ul></ul><ul><li>Cluster 2: </li></ul><ul><ul><li>The shells on the shore are lovely. </li></ul></ul><ul><ul><li>The oyster shell is very hard and black. </li></ul></ul><ul><ul><li>I can hear the ocean in that shell. </li></ul></ul>
  13. 13. Headless Contexts (input) <ul><li>The new version of Linux is more stable and better support for cameras. </li></ul><ul><li>My Chevy Malibu has had some front end troubles. </li></ul><ul><li>Osborne made on of the first personal computers. </li></ul><ul><li>The brakes went out, and the car flew into the house. </li></ul><ul><li>With the price of gasoline, I think I’ll be taking the bus more often! </li></ul>
  14. 14. Headless Contexts (output) <ul><li>Cluster 1: </li></ul><ul><ul><li>The new version of Linux is more stable and better support for cameras. </li></ul></ul><ul><ul><li>Osborne made one of the first personal computers. </li></ul></ul><ul><li>Cluster 2: </li></ul><ul><ul><li>My Chevy Malibu has had some front end troubles. </li></ul></ul><ul><ul><li>The brakes went out, and the car flew into the house. </li></ul></ul><ul><ul><li>With the price of gasoline, I think I’ll be taking the bus more often! </li></ul></ul>
  15. 15. Applications <ul><li>Web search results are headed contexts </li></ul><ul><ul><li>Term you search for is included in snippet </li></ul></ul><ul><li>Web search results are often disorganized – two people sharing same name, two organizations sharing same abbreviation, etc. often have their pages “mixed up” </li></ul><ul><ul><li>Organizing web search results is an important problem. </li></ul></ul><ul><li>If you click on search results or follow links in pages found, you will encounter headless contexts too… </li></ul>
  16. 21. Applications <ul><li>Email (public or private) is made up of headless contexts </li></ul><ul><ul><li>Short, usually focused… </li></ul></ul><ul><li>Cluster similar email messages together </li></ul><ul><ul><li>Automatic email foldering </li></ul></ul><ul><ul><li>Take all messages from sent-mail file or inbox and organize into categories </li></ul></ul>
  17. 24. Applications <ul><li>News article are another example of headless contexts </li></ul><ul><ul><li>Entire article or first paragraph </li></ul></ul><ul><ul><li>Short, usually focused </li></ul></ul><ul><li>Cluster similar articles together </li></ul>
  18. 28. Underlying Premise… <ul><li>You shall know a word by the company it keeps </li></ul><ul><ul><li>Firth, 1957 ( Studies in Linguistic Analysis ) </li></ul></ul><ul><li>Meanings of words are (largely) determined by their distributional patterns (Distributional Hypothesis) </li></ul><ul><ul><li>Harris, 1968 ( Mathematical Structures of Language ) </li></ul></ul><ul><li>Words that occur in similar contexts will have similar meanings (Strong Contextual Hypothesis) </li></ul><ul><ul><li>Miller and Charles, 1991 ( Language and Cognitive Processes ) </li></ul></ul><ul><li>Various extensions… </li></ul><ul><ul><li>Similar contexts will have similar meanings, etc. </li></ul></ul><ul><ul><li>Names that occur in similar contexts will refer to the same underlying person, etc. </li></ul></ul>
  19. 29. Identifying Lexical Features Measures of Association and Tests of Significance
  20. 30. What are features? <ul><li>Features represent the (hopefully) salient characteristics of the contexts to be clustered </li></ul><ul><li>Eventually we will represent each context as a vector, where the dimensions of the vector are associated with features </li></ul><ul><li>Vectors/contexts that include many of the same features will be similar to each other </li></ul>
  21. 31. Where do features come from? <ul><li>In unsupervised clustering, it is common for the feature selection data to be the same data that is to be clustered </li></ul><ul><ul><li>This is not cheating, since data to be clustered does not have any labeled classes that can be used to assist feature selection </li></ul></ul><ul><ul><li>It may also be necessary, since we may need to cluster all available data, and not hold out some for a separate feature identification step </li></ul></ul><ul><ul><ul><li>Email or news articles </li></ul></ul></ul>
  22. 32. Feature Selection <ul><li>“ Test” data – the contexts to be clustered </li></ul><ul><ul><li>Assume that the feature selection data is the same as the test data, unless otherwise indicated </li></ul></ul><ul><li>“ Training” data – a separate corpus of held out feature selection data (that will not be clustered) </li></ul><ul><ul><li>may need to use if you have a small number of contexts to cluster (e.g., web search results) </li></ul></ul><ul><ul><li>This sense of “training” due to Schütze (1998) </li></ul></ul>
  23. 33. Lexical Features <ul><li>Unigram – a single word that occurs more than a given number of times </li></ul><ul><li>Bigram – an ordered pair of words that occur together more often than expected by chance </li></ul><ul><ul><li>Consecutive or may have intervening words </li></ul></ul><ul><li>Co-occurrence – an unordered bigram </li></ul><ul><li>Target Co-occurrence – a co-occurrence where one of the words is the target word </li></ul>
  24. 34. Bigrams <ul><li>fine wine (window size of 2) </li></ul><ul><li>baseball bat </li></ul><ul><li>house of representatives (window size of 3) </li></ul><ul><li>president of the republic (window size of 4) </li></ul><ul><li>apple orchard </li></ul><ul><li>Selected using a small window size (2-4 words), trying to capture a regular (localized) pattern between two words (collocation?) </li></ul>
  25. 35. Co-occurrences <ul><li>tropics water </li></ul><ul><li>boat fish </li></ul><ul><li>law president </li></ul><ul><li>train travel </li></ul><ul><li>Usually selected using a larger window (7-10 words) of context, hoping to capture pairs of related words rather than collocations </li></ul>
  26. 36. Bigrams and Co-occurrences <ul><li>Pairs of words tend to be much less ambiguous than unigrams </li></ul><ul><ul><li>“ bank” versus “river bank” and “bank card” </li></ul></ul><ul><ul><li>“ dot” versus “dot com” and “dot product” </li></ul></ul><ul><li>Three grams and beyond occur much less frequently (Ngrams very Zipfian) </li></ul><ul><li>Unigrams are noisy, but bountiful </li></ul>
  27. 37. “ occur together more often than expected by chance…” <ul><li>Observed frequencies for two words occurring together and alone are stored in a 2x2 matrix </li></ul><ul><ul><li>Throw out bigrams that include one or two stop words </li></ul></ul><ul><li>Expected values are calculated, based on the model of independence and observed values </li></ul><ul><ul><li>How often would you expect these words to occur together, if they only occurred together by chance? </li></ul></ul><ul><ul><li>If two words occur “significantly” more often than the expected value, then the words do not occur together by chance. </li></ul></ul>
  28. 38. 2x2 Contingency Table 100,000 300 !Artificial 400 100 Artificial !Intelligence Intelligence
  29. 39. 2x2 Contingency Table 100,000 99,700 300 99,600 99,400 200 !Artificial 400 300 100 Artificial !Intelligence Intelligence
  30. 40. 2x2 Contingency Table 100,000 99,700 300 99,600 99,400.0 99,301.2 200.0 298.8 !Artificial 400 300.0 398.8 100.0 000.12 Artificial !Intelligence Intelligence
  31. 41. Measures of Association
  32. 42. Measures of Association
  33. 43. Interpreting the Scores… <ul><li>G^2 and X^2 are asymptotically approximated by the chi-squared distribution… </li></ul><ul><li>This means…if you fix the marginal totals of a table, randomly generate internal cell values in the table, calculate the G^2 or X^2 scores for each resulting table, and plot the distribution of the scores, you *should* get … </li></ul>
  34. 45. Interpreting the Scores… <ul><li>Values above a certain level of significance can be considered grounds for rejecting the null hypothesis </li></ul><ul><ul><li>H0: the words in the bigram are independent </li></ul></ul><ul><ul><li>3.841 is associated with 95% confidence that the null hypothesis should be rejected </li></ul></ul>
  35. 46. Measures of Association <ul><li>There are numerous measures of association that can be used to identify bigram and co-occurrence features </li></ul><ul><li>Many of these are supported in the Ngram Statistics Package (NSP) </li></ul><ul><ul><li> </li></ul></ul>
  36. 47. Measures Supported in NSP <ul><li>Log-likelihood Ratio (ll) </li></ul><ul><ul><li>True Mutual Information (tmi) </li></ul></ul><ul><li>Pearson’s Chi-squared Test (x2) </li></ul><ul><li>Pointwise Mutual Information (pmi) </li></ul><ul><li>Phi coefficient (phi) </li></ul><ul><li>T-test (tscore) </li></ul><ul><li>Fisher’s Exact Test (leftFisher, rightFisher) </li></ul><ul><li>Dice Coefficient (dice) </li></ul><ul><li>Odds Ratio (odds) </li></ul>
  37. 48. NSP <ul><li>Will explore NSP during practical session </li></ul><ul><ul><li>Integrated into SenseClusters, may also be used in stand-alone mode </li></ul></ul><ul><li>Can be installed easily on a Linux/Unix system from CD or download from </li></ul><ul><ul><li> </li></ul></ul><ul><ul><li>I’m told it can also be installed on Windows (via cygwin or ActivePerl), but I have no personal experience of this… </li></ul></ul>
  38. 49. Summary <ul><li>Identify lexical features based on frequency counts or measures of association – either in the data to be clustered or in a separate set of feature selection data </li></ul><ul><ul><li>Language independent </li></ul></ul><ul><li>Unigrams usually only selected by frequency </li></ul><ul><ul><li>Remember, no labeled data from which to learn, so somewhat less effective as features than in supervised case </li></ul></ul><ul><li>Bigrams and co-occurrences can also be selected by frequency, or better yet measures of association </li></ul><ul><ul><li>Bigrams and co-occurrences need not be consecutive </li></ul></ul><ul><ul><li>Stop words should be eliminated </li></ul></ul><ul><ul><li>Frequency thresholds are helpful (e.g., unigram/bigram that occurs once may be too rare to be useful) </li></ul></ul>
  39. 50. Related Work <ul><li>Moore, 2004 (EMNLP) follow-up to Dunning and Pedersen on log-likelihood and exact tests </li></ul><ul><li> </li></ul><ul><li>Pedersen, 1996 (SCSUG) explanation of exact tests, and comparison to log-likelihood </li></ul><ul><li> </li></ul><ul><li>(also see Pedersen, Kayaalp, and Bruce, AAAI-1996) </li></ul><ul><li>Dunning, 1993 ( Computational Linguistics ) introduces log-likelihood ratio for collocation identification </li></ul><ul><li> </li></ul>
  40. 51. Context Representations First and Second Order Methods
  41. 52. Once features selected… <ul><li>We will have a set of unigrams, bigrams, co-occurrences or target co-occurrences that we believe are somehow interesting and useful </li></ul><ul><ul><li>We also have any frequency and measure of association score that have been used in their selection </li></ul></ul><ul><li>Convert contexts to be clustered into a vector representation based on these features </li></ul>
  42. 53. First Order Representation <ul><li>Each context is represented by a vector with M dimensions, each of which indicates whether or not a particular feature occurred in that context </li></ul><ul><ul><li>Value may be binary, a frequency count, or an association score </li></ul></ul><ul><li>Context by Feature representation </li></ul>
  43. 54. Contexts <ul><li>C1: There was an island curse of black magic cast by that voodoo child. </li></ul><ul><li>C2: Harold, a known voodoo child, was gifted in the arts of black magic. </li></ul><ul><li>C3: Despite their military might, it was a serious error to attack. </li></ul><ul><li>C4: Military might is no defense against a voodoo child or an island curse. </li></ul>
  44. 55. Unigram Feature Set <ul><li>island 1000 </li></ul><ul><li>black 700 </li></ul><ul><li>curse 500 </li></ul><ul><li>magic 400 </li></ul><ul><li>child 200 </li></ul><ul><li>(assume these are frequency counts obtained from some corpus…) </li></ul>
  45. 56. First Order Vectors of Unigrams 1 0 1 0 1 C4 0 0 0 0 0 C3 1 1 0 1 0 C2 1 1 1 1 1 C1 child magic curse black island
  46. 57. Bigram Feature Set <ul><li>island curse 189.2 </li></ul><ul><li>black magic 123.5 </li></ul><ul><li>voodoo child 120.0 </li></ul><ul><li>military might 100.3 </li></ul><ul><li>serious error 89.2 </li></ul><ul><li>island child 73.2 </li></ul><ul><li>voodoo might 69.4 </li></ul><ul><li>military error 54.9 </li></ul><ul><li>black child 43.2 </li></ul><ul><li>serious curse 21.2 </li></ul><ul><li>(assume these are log-likelihood scores based on frequency counts from some corpus) </li></ul>
  47. 58. First Order Vectors of Bigrams 1 0 1 1 0 C4 0 1 1 0 0 C3 1 0 0 0 1 C2 1 0 0 1 1 C1 voodoo child serious error military might island curse black magic
  48. 59. First Order Vectors <ul><li>Can have binary values or weights associated with frequency, etc. </li></ul><ul><li>May optionally be smoothed/reduced with Singular Value Decomposition </li></ul><ul><ul><li>More on that later… </li></ul></ul><ul><li>The contexts are ready for clustering… </li></ul><ul><ul><li>More on that later… </li></ul></ul>
  49. 60. Second Order Representation <ul><li>Build word by word matrix from features </li></ul><ul><ul><li>Must be bigrams or co-occurrences </li></ul></ul><ul><ul><li>(optionally) reduce dimensionality w/SVD </li></ul></ul><ul><ul><li>Each row represents first order co-occurrences </li></ul></ul><ul><li>Represent a context by replacing each word with an entry in the word by word matrix with its associated vector </li></ul><ul><li>Average word vectors found for the context </li></ul><ul><li>Due to Schuetze (1998) </li></ul>
  50. 61. Word by Word Matrix 120.0 0 69.4 0 0 voodoo 0 89.2 0 21.2 0 serious 0 54.9 100.3 0 0 military 73.2 0 0 189.2 0 island 43.2 0 0 0 123.5 black child error might curse magic
  51. 62. Word by Word Matrix <ul><li>… can also be used to identify sets of related words </li></ul><ul><li>In the case of bigrams, rows represent the first word in a bigram and columns represent the second word </li></ul><ul><ul><li>Matrix is asymmetric </li></ul></ul><ul><li>In the case of co-occurrences, rows and columns are equivalent </li></ul><ul><ul><li>Matrix is symmetric </li></ul></ul><ul><li>The vector (row) for each word represent a set of first order features for that word </li></ul><ul><li>Each word in a context to be clustered for which a vector exists (in the word by word matrix) is replaced by that vector in that context </li></ul>
  52. 63. There was an island curse of black magic cast by that voodoo child. 120.0 0 69.4 0 0 voodoo 73.2 0 0 189.2 0 island 43.2 0 0 0 123.5 black child error might curse magic
  53. 64. Second Order Representation <ul><li>There was an [curse, child] curse of [magic, child] magic cast by that [might, child] child </li></ul><ul><li>[curse, child] + [magic, child] + [might, child] </li></ul>
  54. 65. There was an island curse of black magic cast by that voodoo child. 78.8 0 24.4 63.1 41.2 C1 child error might curse magic
  55. 66. First versus Second Order <ul><li>First Order represents a context by showing which features occurred in that context </li></ul><ul><ul><li>This is what feature vectors normally do… </li></ul></ul><ul><li>Second Order allows for additional information about a word to be incorporated into the representation </li></ul><ul><ul><li>Feature values based on information found outside of the immediate context </li></ul></ul>
  56. 67. Second Order Co-Occurrences <ul><li>“black” and “island” show similarity because both words have occurred with “child” </li></ul><ul><li>“black” and “island” are second order co-occurrence with each other, since both occur with “child” but not with each other (i.e., “black island” is not observed) </li></ul>
  57. 68. Second Order Co-occurrences <ul><li>Imagine a co-occurrence graph </li></ul><ul><ul><li>Word network </li></ul></ul><ul><li>First order co-occurrences are directly connected </li></ul><ul><li>Second order co-occurrences are to each connected via one other word </li></ul><ul><li> program in Ngram Statistics Package finds kth order co-occurrences </li></ul>
  58. 69. Summary <ul><li>First order representations are intuitive, but… </li></ul><ul><ul><li>Can suffer from sparsity </li></ul></ul><ul><ul><li>Contexts represented based on the features that occur in those contexts </li></ul></ul><ul><li>Second order representations are harder to visualize, but… </li></ul><ul><ul><li>Allow a word to be represented by the words it co-occurs with (i.e., the company it keeps) </li></ul></ul><ul><ul><li>Allows a context to be represented by the words that occur with the words in the context </li></ul></ul><ul><ul><li>Helps combat sparsity… </li></ul></ul>
  59. 70. Related Work <ul><li>Pedersen and Bruce 1997 (EMNLP) presented first order method of discrimination </li></ul><ul><li> </li></ul><ul><li>Schütze 1998 ( Computational Linguistics ) introduced second order method </li></ul><ul><li> </li></ul><ul><li>Purandare and Pedersen 2004 (CoNLL) compared first and second order methods </li></ul><ul><li> </li></ul><ul><ul><li>First order better if you have lots of data </li></ul></ul><ul><ul><li>Second order better with smaller amounts of data </li></ul></ul>
  60. 71. Dimensionality Reduction Singular Value Decomposition
  61. 72. Motivation <ul><li>First order matrices are very sparse </li></ul><ul><ul><li>Word by word </li></ul></ul><ul><ul><li>Context by feature </li></ul></ul><ul><li>NLP data is noisy </li></ul><ul><ul><li>No stemming performed </li></ul></ul><ul><ul><li>synonyms </li></ul></ul>
  62. 73. Many Methods <ul><li>Singular Value Decomposition (SVD) </li></ul><ul><ul><li>SVDPACKC http:// / </li></ul></ul><ul><li>Multi-Dimensional Scaling (MDS) </li></ul><ul><li>Principal Components Analysis (PCA) </li></ul><ul><li>Independent Components Analysis (ICA) </li></ul><ul><li>Linear Discriminant Analysis (LDA) </li></ul><ul><li>etc… </li></ul>
  63. 74. Effect of SVD <ul><li>SVD reduces a matrix to a given number of dimensions This may convert a word level space into a semantic or conceptual space </li></ul><ul><ul><li>If “dog” and “collie” and “wolf” are dimensions/columns in a word co-occurrence matrix, after SVD they may be a single dimension that represents “canines” </li></ul></ul>
  64. 75. Effect of SVD <ul><li>The dimensions of the matrix after SVD are principal components that represent the meaning of concepts </li></ul><ul><ul><li>Similar columns are grouped together </li></ul></ul><ul><li>SVD is a way of smoothing a very sparse matrix, so that there are very few zero valued cells after SVD </li></ul>
  65. 76. How can SVD be used? <ul><li>SVD on first order contexts will reduce a context by feature representation down to a smaller number of features </li></ul><ul><ul><li>Latent Semantic Analysis typically performs SVD on a word by context representation, where the contexts are reduced </li></ul></ul><ul><li>SVD used in creating second order context representations </li></ul><ul><ul><li>Reduce word by word matrix </li></ul></ul><ul><li>SVD could also be used on resultant second order context representations (although not supported) </li></ul>
  66. 77. Word by Word Matrix 4 2 0 0 0 3 0 1 box 0 1 2 2 1 2 0 0 memory 0 0 0 1 0 0 2 0 organ 0 2 0 3 2 0 0 0 debt 0 1 0 3 1 0 0 2 linux 0 1 0 3 2 0 0 0 sales 3 0 2 2 0 3 0 0 lab 1 0 2 0 0 1 2 0 petri 0 1 0 0 2 0 0 1 disk 1 0 2 0 0 0 3 0 body 0 0 0 3 1 0 0 2 pc plasma graphics tissue data ibm cells blood apple
  67. 78. Singular Value Decomposition A=UDV’
  68. 79. U -.52 .39 -.48 .02 .09 .41 -.09 .40 -.30 .08 .31 .43 -.26 -.39 -.6 .20 .00 -.00 -.00 -.02 -.01 .00 -.02 -.00 -.07 -.3 .14 -.49 -.07 .30 .25 .56 -.01 .08 .05 -.01 .24 -.08 .11 .46 .08 .03 -.04 .72 .09 -.31 -.01 .37 -.07 .01 -.21 -.31 -.34 -.45 -.68 .29 .00 .05 .83 .17 -.02 .25 -.45 .08 .03 .20 -.22 .31 -.60 .39 .13 .35 -.01 -.04 -.44 .08 .44 .59 -.49 .05 -.02 .63 .02 -.09 .52 -.2 .09 .35
  69. 80. D 0.00 0.00 0.00 0.66 1.26 2.30 2.52 3.25 3.99 6.36 9.19
  70. 81. V -.20 .22 -.07 -.10 -.87 -.07 -.06 .17 .19 -.26 .04 .03 .17 -.32 .02 .13 -.26 -.17 .06 -.04 .86 .50 -.58 .12 .09 -.18 -.27 -.18 -.12 -.47 .11 -.03 .12 .31 -.32 -.04 .64 -.45 -.14 -.23 .28 .07 -.23 -.62 -.59 .05 .02 -.12 .15 .11 .25 -.71 -.31 -.04 .08 .29 -.05 .05 .20 -.51 .09 -.03 .12 .31 -.01 .02 -.45 -.32 .50 .27 .49 -.02 .08 .21 -.06 .08 -.09 .52 -.45 -.01 .63 .03 -.12 -.31 .71 -.13 .39 -.12 .12 .15 .37 .07 .58 -.41 .15 .17 -.30 -.32 -.27 -.39 .11 .44 .25 .03 -.02 .26 .23 .39 .57 -.37 .04 .03 -.12 -.31 -.05 -.05 .04 .28 -.04 .08 .21
  71. 82. Word by Word Matrix After SVD 1.1 1.0 .98 1.7 .86 .72 .85 .77 memory .00 .00 .17 1.2 .77 .00 .84 .00 organ .00 1.5 .00 3.2 2.1 .00 .00 1.2 debt .13 1.1 .03 2.7 1.7 .16 .00 .96 linux .41 .85 .35 2.2 1.3 .39 .15 .73 sales 2.3 .18 2.5 1.7 .35 2.0 1.7 .21 lab 1.4 .00 1.5 .49 .00 1.2 1.1 .00 germ .00 .91 .00 2.1 1.3 .01 .00 .76 disk 1.5 .00 1.6 .33 .00 1.3 1.2 .00 body .09 .86 .01 2.0 1.3 .11 .00 .73 pc plasma graphics tissue data ibm cells blood apple
  72. 83. Second Order Representation <ul><li>These two contexts share no words in common, yet they are similar! disk and linux both occur with “Apple”, “IBM”, “data”, “graphics”, and “memory” </li></ul><ul><li>The two contexts are similar because they share many second order co-occurrences </li></ul><ul><li>I got a new disk today! </li></ul><ul><li>What do you think of linux? </li></ul>1.0 .72 memory .00 .00 organ .13 1.1 .03 2.7 1.7 .16 .00 .96 linux .00 .91 .00 2.1 1.3 .01 .00 .76 disk Plasma graphics tissue data ibm cells blood apple
  73. 84. Clustering Methods Agglomerative and Partitional
  74. 85. Many many methods… <ul><li>Cluto supports a wide range of different clustering methods </li></ul><ul><ul><li>Agglomerative </li></ul></ul><ul><ul><ul><li>Average, single, complete link… </li></ul></ul></ul><ul><ul><li>Partitional </li></ul></ul><ul><ul><ul><li>K-means </li></ul></ul></ul><ul><ul><li>Hybrid </li></ul></ul><ul><ul><ul><li>Repeated bisections </li></ul></ul></ul><ul><li>SenseClusters integrates with Cluto </li></ul><ul><ul><li> </li></ul></ul>
  75. 86. General Methodology <ul><li>Represent contexts to be clustered in first or second order vectors </li></ul><ul><li>Cluster the vectors directly, or convert to similarity matrix and then cluster </li></ul><ul><ul><li>vcluster </li></ul></ul><ul><ul><li>scluster </li></ul></ul>
  76. 87. Agglomerative Clustering <ul><li>Create a similarity matrix of instances to be discriminated </li></ul><ul><ul><li>Results in a symmetric “instance by instance” matrix, where each cell contains the similarity score between a pair of instances </li></ul></ul><ul><ul><li>Typically a first order representation, where similarity is based on the features observed in the pair of instances </li></ul></ul>
  77. 88. Measuring Similarity <ul><li>Integer Values </li></ul><ul><ul><li>Matching Coefficient </li></ul></ul><ul><ul><li>Jaccard Coefficient </li></ul></ul><ul><ul><li>Dice Coefficient </li></ul></ul><ul><li>Real Values </li></ul><ul><ul><li>Cosine </li></ul></ul>
  78. 89. Agglomerative Clustering <ul><li>Apply Agglomerative Clustering algorithm to similarity matrix </li></ul><ul><ul><li>To start, each instance is its own cluster </li></ul></ul><ul><ul><li>Form a cluster from the most similar pair of instances </li></ul></ul><ul><ul><li>Repeat until the desired number of clusters is obtained </li></ul></ul><ul><li>Advantages : high quality clustering </li></ul><ul><li>Disadvantages – computationally expensive, must carry out exhaustive pair wise comparisons </li></ul>
  79. 90. Average Link Clustering 1 2 4 S3 1 2 4 S3 0 2 S4 0 3 S2 2 3 S1 S4 S2 S1 0 S4 0 S2 S1S3 S4 S2 S1S3 S4 S1S3S2 S4 S1S3S2
  80. 91. Partitional Methods <ul><li>Select some number of contexts in feature space to act as centroids </li></ul><ul><li>Assign each context to nearest centroid, forming cluster </li></ul><ul><li>After all contexts assigned, recompute centroids </li></ul><ul><li>Repeat until stable clusters found </li></ul><ul><ul><li>Centroids don’t shift from iteration to iteration </li></ul></ul>
  81. 92. Partitional Methods <ul><li>Advances : fast </li></ul><ul><li>Disadvantages : very dependent on the initial placement of centroids </li></ul>
  82. 93. Cluster Labeling
  83. 94. Results of Clustering <ul><li>Each cluster consists of some number of contexts </li></ul><ul><li>Each context is a short unit of text </li></ul><ul><li>Apply measures of association to the contents of each cluster to determine N most significant bigrams </li></ul><ul><li>Use those bigrams as a label for the cluster </li></ul>
  84. 95. Label Types <ul><li>The N most significant bigrams for each cluster will act as a descriptive label </li></ul><ul><li>The M most significant bigrams that are unique to each cluster will act as a discriminating label </li></ul>
  85. 96. Evaluation Techniques Comparison to gold standard data
  86. 97. Evaluation <ul><li>If Sense tagged text is available, can be used for evaluation </li></ul><ul><ul><li>But don’t use sense tags for clustering or feature selection! </li></ul></ul><ul><li>Assume that sense tags represent “true” clusters, and compare these to discovered clusters </li></ul><ul><ul><li>Find mapping of clusters to senses that attains maximum accuracy </li></ul></ul>
  87. 98. Evaluation <ul><li>Pseudo words are especially useful, since it is hard to find data that is discriminated </li></ul><ul><ul><li>Pick two words or names from a corpus, and conflate them into one name. Then see how well you can discriminate. </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>Baseline Algorithm– group all instances into one cluster, this will reach “accuracy” equal to majority classifier </li></ul>
  88. 99. Evaluation <ul><li>Pseudo words are especially useful, since it is hard to find data that is discriminated </li></ul><ul><ul><li>Pick two words or names from a corpus, and conflate them into one name. Then see how well you can discriminate. </li></ul></ul><ul><ul><li> </li></ul></ul>
  89. 100. Baseline Algorithm <ul><li>Baseline Algorithm – group all instances into one cluster, this will reach “accuracy” equal to majority classifier </li></ul><ul><li>What if the clustering said everything should be in the same cluster? </li></ul>
  90. 101. Baseline Performance <ul><li>(0+0+55)/170 = .32 if C3 is S1 (0+0+80)/170 = .47 if C3 is S3 </li></ul>170 55 35 80 Totals 170 55 35 80 C3 0 0 0 0 C2 0 0 0 0 C1 Totals S3 S2 S1 170 80 35 55 Totals 170 80 35 55 C3 0 0 0 0 C2 0 0 0 0 C1 Totals S1 S2 S3
  91. 102. Evaluation <ul><li>Suppose that C1 is labeled S1, C2 as S2, and C3 as S3 </li></ul><ul><li>Accuracy = (10 + 0 + 10) / 170 = 12% </li></ul><ul><li>Diagonal shows how many members of the cluster actually belong to the sense given on the column </li></ul><ul><li>Can the “columns” be rearranged to improve the overall accuracy? </li></ul><ul><ul><li>Optimally assign clusters to senses </li></ul></ul>170 55 35 80 Totals 65 10 5 50 C3 60 40 0 20 C2 45 5 30 10 C1 Totals S3 S2 S1
  92. 103. Evaluation <ul><li>The assignment of C1 to S2, C2 to S3, and C3 to S1 results in 120/170 = 71% </li></ul><ul><li>Find the ordering of the columns in the matrix that maximizes the sum of the diagonal. </li></ul><ul><li>This is an instance of the Assignment Problem from Operations Research, or finding the Maximal Matching of a Bipartite Graph from Graph Theory. </li></ul>170 80 55 35 Totals 65 50 10 5 C3 60 20 40 0 C2 45 10 5 30 C1 Totals S1 S3 S2
  93. 104. Analysis <ul><li>Unsupervised methods may not discover clusters equivalent to the classes learned in supervised learning </li></ul><ul><li>Evaluation based on assuming that sense tags represent the “true” cluster are likely a bit harsh. Alternatives? </li></ul><ul><ul><li>Humans could look at the members of each cluster and determine the nature of the relationship or meaning that they all share </li></ul></ul><ul><ul><li>Use the contents of the cluster to generate a descriptive label that could be inspected by a human </li></ul></ul>
  94. 105. Practical Session Experiments with SenseClusters
  95. 106. Experimental Data <ul><li>Available on Web Site </li></ul><ul><ul><li> </li></ul></ul><ul><li>Available on CD </li></ul><ul><ul><li>Data/SenseClusters-Data </li></ul></ul><ul><li>SenseClusters requires data to be in the Senseval-2 lexical sample format </li></ul><ul><ul><li>Plenty of such data available on CD and from web site </li></ul></ul>
  96. 107. Creating Experimental Data <ul><li>NameConflate program </li></ul><ul><ul><li>Creates name conflated data from English GigaWord corpus </li></ul></ul><ul><li>Text2Headless program </li></ul><ul><ul><li>Convert plain text into headless contexts </li></ul></ul><ul><li>http:// </li></ul>
  97. 108. Name Conflation Data <ul><li>Smaller Data Set (also on Web as SC-Web…) </li></ul><ul><ul><li>Country - Noun </li></ul></ul><ul><ul><li>Name - Name </li></ul></ul><ul><ul><li>Noun - Noun </li></ul></ul><ul><li>Larger Data Sets (also on Web as Split-Smaller…) </li></ul><ul><ul><li>Adidas - Puma </li></ul></ul><ul><ul><li>Emile Lahoud – Askar Akayev </li></ul></ul><ul><li>CICLING data (CD only) </li></ul><ul><ul><li>David Beckham – Ronaldo </li></ul></ul><ul><ul><li>Microsoft – IBM </li></ul></ul><ul><li>ACL 2005 demo data (CD only) </li></ul><ul><ul><li>Name - Name </li></ul></ul>
  98. 109. Clustering Contexts <ul><li>ACL 2005 Demo (also on Web as Email…) </li></ul><ul><ul><li>Various partitions of 20 news groups data sets </li></ul></ul><ul><li>Spanish Data (web only) </li></ul><ul><ul><li>News articles each of which mention abbreviations PP or PSOE </li></ul></ul>
  99. 110. Name Discrimination
  100. 111. George Millers!
  101. 112. Headed Clustering <ul><li>Name Discrimination </li></ul><ul><ul><li>Tom Hanks </li></ul></ul><ul><ul><li>Russell Crowe </li></ul></ul>
  102. 135. Headless Contexts <ul><li>Email / 20 newsgroups data </li></ul><ul><li>Spanish Text </li></ul>
  103. 157. If you after all these matrices you crave knowledge based resources… Read on…
  104. 158. WordNet-Similarity <ul><li>Not language independent </li></ul><ul><ul><li>Based on English WordNet </li></ul></ul><ul><li>But, can be combined with distributional methods to good effect </li></ul><ul><ul><li>McCarthy, et. al. ACL-2004 </li></ul></ul><ul><li>Perl module </li></ul><ul><ul><li>http:// /dist/WordNet-Similarity </li></ul></ul><ul><li>Web interface </li></ul><ul><ul><li> </li></ul></ul>
  105. 159. Many thanks! <ul><li>Satanjeev “Bano” Banerjee (M.S., 2002) </li></ul><ul><ul><li>Inventor of Adapted Lesk Algorithm (IJCAI-2003), which is the earliest origin and motivation for WordNet-Similarity… </li></ul></ul><ul><ul><li>Now PhD student at LTI/CMU… </li></ul></ul><ul><li>Siddharth Patwardhan (M.S., 2003) </li></ul><ul><ul><li>Founding developer of WordNet-Similarity (2001-2003) </li></ul></ul><ul><ul><li>Now PhD student at University of Utah </li></ul></ul><ul><ul><li>http:// / </li></ul></ul><ul><li>Jason Michelizzi (M.S., 2005) </li></ul><ul><ul><li>Enhanced WordNet-Similarity in many ways and applied it to all words sense disambiguation (2003-2005) </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>NSF for supporting Bano, and University of Minnesota for supporting Bano, Sid and Jason via various internal sources </li></ul>
  106. 160. Vector measure <ul><li>Build a word by word matrix from WordNet Gloss Corpus </li></ul><ul><ul><li>1.4 million words </li></ul></ul><ul><li>Treat glosses as contexts, and use second order representation where words are replaced with vectors from matrix </li></ul><ul><ul><li>Average together all vectors to represent concept/definition </li></ul></ul><ul><li>High correlation with human relatedness judgements </li></ul>
  107. 161. Many other measures <ul><li>Path Based </li></ul><ul><ul><li>Path </li></ul></ul><ul><ul><li>Leacock & Chodorow </li></ul></ul><ul><ul><li>Wu and Palmer </li></ul></ul><ul><li>Information Content Based </li></ul><ul><ul><li>Resnik </li></ul></ul><ul><ul><li>Lin </li></ul></ul><ul><ul><li>Jiang & Conrath </li></ul></ul><ul><li>Relatedness </li></ul><ul><ul><li>Hirst & St-Onge </li></ul></ul><ul><ul><li>Adapted Lesk </li></ul></ul><ul><ul><li>Vector </li></ul></ul>
  108. 164. Thank you! <ul><li>Questions are welcome at any time. Feel free to contact me in person or via email ( [email_address] ) at any time! </li></ul><ul><li>All of our software is free and open source, you are welcome to download, modify, redistribute, etc. </li></ul><ul><ul><li> </li></ul></ul>