• Like
Similarity  Measurement  Preliminary Results
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Similarity Measurement Preliminary Results



Published in Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • Spearman’s rank correlation


  • 1. Similarity measurement:Folksonomyvs.LSA
    Preliminary Results
  • 2. The Tripartite structure of tagging
    Folksonomy is a set of triples < user , tag, object>
    A folksonomy is a tuple F :=(U, T, R, Y) where U, T, and R are finite sets, whose elements are called users, tags and resources. Y is a ternary relation between user, tags and resources.
  • 3. Del.icio.us Tag distribution
    Tag distribution
    Log-log Tag distribution
    After crawling the delicious.com site, the total number tags (tokens) obtained was 7,528,528, among which the number of types was 188,964. All the tags are stemmed using the Porter Stemmer and the total number of stemmed tags ended up to be 174,887.
  • 4. LSA Processing Workflow in R
    tm = textmatrix(‘dir/‘)
    tm = lw_logtf(tm) * gw_idf(tm)
    space = lsa(tm, dims=dimcalc_share())
  • 5. LSA corpus preparing
    A total number of 17,085 web pages were crawled and were later parsed to remove all the HTML markups.
    Stemming and Stop-word removal
    The processed corpus:14,993,620 tokens, 259,464 types of words.
    Only words with frequency more than 100 were kept to be entered into a word-by-document matrix. There were 1047 words with frequency more than 100. The text length kept in the corpus is from ranges from 51 words per document to 4009 words per document .
    Therefore, the resulting term-document matrix have 3465 columns (documents) and 1082 rows (words). 
  • 6. LSA document length distribution
    The text length kept in the corpus is from ranges from 51 words per document to 4009 words per document .
  • 7. Three similarity measurements
    Tag Co-coccurrence counts
    Tag vector cosine similarity
  • 8. Similarity Measurement
    Tag Co-occurrence Counts:
    1)simple count: how many times two tags are used by the same user to annotate the same resource
    2)normalized count: Jaccard IndexThe co-occurrence counts of tag A and tag B divided by the joint frequency of A and B.
  • 9. Distribution of Tag co-occurrence Counts (simple counts)
  • 10. Distribution of Tag co-occurrence Counts (normalized)
  • 11. Measurement 2: Cosine Similarity
    Based on the co-occurrence vector of each tag with every other tag.
    Since there are normalized and unnormalized tag co-occurrence counts, we then ended up with two
    X and Y are the co-occurrence vectors of two distinct tags.
  • 12. Distribution of Tag Cosine Similarity
  • 13. Distribution of Tag Cosine Similarity based on normalized Tag Co-occurrence counts
  • 14. Results
    The pair-wise Pearson correlation and Spearman correlation among 5 measurements [Tag-cooccurrence count, Tag cosine similarity, LSA, normalized Tag-cooccurrence count, Tag cosine similarity based on normalized tag-tag cooccurrence matrix]
  • 15. Correlation
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21. Qualitative Insight – ‘linguistics’
    Top 10 “linguistics” related words according to 5 measurements
  • 22. Correlation between two measurements:normalized tag co-occurrence counts vs. normalized tag cosine
    P= 0.7547