Pedersen semeval-2013-poster-may24


Published on

Poster presentation from SemEval-2013 workshop regarding Task 11, a shared task on performing word sense induction within an end-user application.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Pedersen semeval-2013-poster-may24

  1. 1. Poster Design & Printing by Genigraphics®- 800.790.4001Ted PedersenDepartment of Computer ScienceUniversity of Minnesota, Duluthtpederse@d.umn.edu PedersenUniversity of Minnesota, Duluthhttp://senseclusters.sourceforge.netSystems use different corpora to build co-occurrence matrix, otherwise nearly identical.Sys7 : Uses smallest corpora, just the 64snippets per term. Resulting matrices rangefrom 102 X 113 to 221 X 222. No SVD.Sys1: Use all 6400 snippets to create matrixof size 771 X 952. SVD reduced to 771 X 90.Sys9: Use first 10,000 paragraphs of APW datafrom English Gigaword. Resulting matrix is9,853 X 10,995. SVD reduced to 9,853 X 300.RandX: a random baseline, assign each Websnippet to one of X random senses. Evaluationmeasures should be able to expose randombaselines and give them appropriately lowscores. Both the paired F-score used inSemEval-2010 and the Jaccard Coefficientsatisfy this requirement.Given 64 snippets per ambiguous term,first order features were unlikely tosucceed and achieved F-10 score of 36.10According to F-10, F-SC, and Jaccard,smaller amounts of task specific data(sys7 and sys1) are more effective thanlarge amounts of out of domain text (sys9)when using second order methods.Newspaper text (like APW as used in Sys9)is not typically what Web search locates.Results are more often commercial,Wikipedia, or current celebrity.Lessons learned? :●For second order methods, use Web-likedata, not news text●Use more data, increase snippets withWeb, Wikipedia, etc and then discardadditions after clustering (?)●Expand snippets by going to site in result•Task 11 goal? Cluster Web search results!Test Data? The top 64 Google results foreach of 100 potentially ambiguous queries.Each result is a Web snippet about 25words long.Challenges? Small amounts of data!• 64 results/term X 25 words/result =1,600 words/term X 100 terms =160,000 wordsSolution? Augment the data!• Use second order co-occurrences toenrich Web snippets to be clustered, friendof a friend relation• The bigrams car motor, car insurance,car magazine, life sentence, life force, andlife insurance each represent a first orderco-occurrence. Car and life are secondorder co-occurrences because both occurwith insurance.IntroductionGeneric Duluth SystemConclusion and Future DirectionsDiscussionThe Duluth SystemsAbstractContactDuluth : Word Sense Induction Applied to Web Page ClusteringTask 11 : Word Sense Induction & Disambiguation within an End-User ApplicationExperimental ResultsSystem F-102010Jaccard F1-132014ARI Clusters/SizeSys1 46.53 31.79 56.83 5.752.5/26.5Sys7 45.89 31.03 58.78 6.783.0/25.2Sys9 35.56 22.24 57.02 2.593.3/19.8Rand2 41.49 26.99 54.89 –0.042.0/32.0Rand5 25.17 14.52 56.73 0.125.0/12.8Rand10 15.05 8.18 59.67 0.0210.0/6.4Rand25 7.01 3.63 66.89? -0.1523.2/2.8Rand50 4.07 2.00 76.19? 0.1035.9/1.8MFS 54.06 39.90 54.42 0.00?1.0/64.0Gold 100.00 100.00 100.00 99.007.7/11.6The Duluth systems thatparticipated in Task 11 ofSemEval–2013 carried outword sense induction (WSI)in order to cluster Websearch results. They reliedon an approach thatrepresented Web snippetsusing second-order co-occurrences. Thesesystems were allimplemented usingSenseClusters, a freelyavailable open sourcesoftware package.Web page clustering viewed as a wordsense discrimination / induction problemwhere query term is the target word,snippet provides surrounding context•Create co-occurrence matrix from somecorpus. Rows and columns made up of firstand second word of bigrams identified bylog-likelihood ratio. Low frequency and lowscoring bigrams are excluded, as are stopwords. Optionally reduce dimensionalitywith SVD.•Replace each word in a web snippet witha vector made up of its row from the co-occurrence matrix. Average all vectors fora snippet together to create a new contextvector. This will capture second order co-occurrences between context vectors.•Cluster the resulting 64 vectors for anambiguous term, find number of clustersautomatically with PK2.Average number of senses is 7.7, butvariance is quite high ...•heron island – 1 sense, 100% MFS•Shakira – 2 senses, 98% MFS•apple – 2 senses, 98% MFS•kawasaki – 7 senses, 47% MFS•Billy the Kid – 7 senses, 44% MFS•marble – 8 senses, 39% MFS•kangaroo – 17 senses, 48% MFS•ghost – 18 senses, 30% MFS•dog eat dog – 19 senses, 28% MFS•magic – 19 senses, 27% MFSOf 769 senses in the test data ...•467 (61%) occur less than 5 times!!• Is 1, 2, 3, 4, … instances enough toidentify a cluster?• Very small clusters often “pure”, cantrick some evaluation methods•186 (24%) are defined as “Other”• Criteria for membership unclear ordifferent than other senses• Other different than cant cluster?